Source-linked AI summary
TDAN: Temporally Deformable Alignment Network for Video Super-Resolution
Yapeng Tian, Yulun Zhang, Yun Fu, Chenliang Xu
TL;DR
VSR must align motion-displaced supporting frames with a reference frame, while optical-flow wrapping can introduce artifacts. TDAN performs feature-level deformable alignment without optical flow, and its VSR framework achieves state-of-the-art results on Vid4 and reported benchmark and real-sequence evaluations.
Problem
VSR needs to align motion-displaced supporting frames with the reference frame, but optical-flow estimation and image-level wrapping can introduce artifacts.
Method
TDAN uses reference and supporting-frame features to predict deformable sampling offsets for one-stage feature-level alignment without optical flow, followed by SR reconstruction.
Results
The TDAN-based framework achieves state-of-the-art VSR performance on the Vid4 benchmark and demonstrates effectiveness in extensive benchmark and real-world sequence experiments.
Takeaways & Limitations
TDAN can exploit supporting-frame information through feature-level temporal alignment without explicit motion estimation or image-wise wrapping.
Takeaways & Limitations
A reported failure case shows TDAN does not recover some building structures that a very deep DIV2K-trained RCAN reconstructs accurately, motivating larger high-resolution VSR datasets.
Abstract
from arXiv · showhide
Video super-resolution (VSR) aims to restore a photo-realistic high-resolution (HR) video frame from both its corresponding low-resolution (LR) frame (reference frame) and multiple neighboring frames (supporting frames). Due to varying motion of cameras or objects, the reference frame and each support frame are not aligned. Therefore, temporal alignment is a challenging yet important problem for VSR. Previous VSR methods usually utilize optical flow between the reference frame and each supporting frame to wrap the supporting frame for temporal alignment. Therefore, the performance of these image-level wrapping-based models will highly depend on the prediction accuracy of optical flow, and inaccurate optical flow will lead to artifacts in the wrapped supporting frames, which also will be propagated into the reconstructed HR video frame. To overcome the limitation, in this paper, we propose a temporal deformable alignment network (TDAN) to adaptively align the reference frame and each supporting frame at the feature level without computing optical flow. The TDAN uses features from both the reference frame and each supporting frame to dynamically predict offsets of sampling convolution kernels. By using the corresponding kernels, TDAN transforms supporting frames to align with the reference frame. To predict the HR video frame, a reconstruction network taking aligned frames and the reference frame is utilized. Experimental results demonstrate the effectiveness of the proposed TDAN-based VSR model.
1. Introduction
VSR reconstructs an HR frame from a reference LR frame and neighboring supporting frames, but motion-induced misalignment makes temporal alignment central. TDAN addresses this challenge with feature-level deformable alignment and achieves state-of-the-art performance on the Vid4 benchmark.
- Motivation: VSR reconstructs an HR video frame from a corresponding LR reference frame and multiple neighboring LR supporting frames.HR frames contain more image details and are desired in applications including video surveillance and HDTV.
- Problem: Camera or object motion can leave the reference and supporting frames misaligned, making supporting-frame alignment a vital VSR issue.
- Limitations of prior methods: Optical-flow methods estimate motion fields and wrap supporting frames, but flow or image-level wrapping errors may introduce artifacts around image structures.
- Proposed method: TDAN performs one-stage feature-level temporal alignment without optical flow, using reference and supporting features to predict deformable sampling offsets.
- Results: TDAN-based VSR achieves state-of-the-art performance on the Vid4 benchmark and reconstructs more image details than the recent DUF network in a walk-sequence comparison.
2. Related Work
Prior work spans SISR, optical-flow-based VSR, and deformable convolution. TDAN extends deformable alignment toward feature-level VSR while using multiple neighboring frames in its framework.
- Single Image Super-Resolution (SISR): SISR progressed from SRCNN and VDSR to sub-pixel upscaling and deeper architectures such as EDSR, RDN, DBPN, and RCAN.
- Video Super-Resolution (VSR): VSR methods commonly use two stages: optical-flow motion estimation followed by image wrapping or motion compensation.
- Video Super-Resolution (VSR): End-to-end VSR methods jointly train flow estimation and spatiotemporal processing, while others adapt temporal dependence or perform sub-pixel motion compensation.
- Deformable Convolution: Deformable convolution improves modeling of geometric transformations beyond fixed CNN kernels and has been applied mainly to high-level vision tasks.
- Proposed framework: The TDAN-based framework exploits four neighboring supporting frames in implementation to explore additional temporal information.
3. Method
TDAN aligns supporting video frames with a reference frame at the feature level, using deformable sampling offsets rather than optical flow. Aligned frames are reconstructed and jointly optimized in an end-to-end VSR framework.
- 3.1. Overview: The framework processes consecutive 2N + 1 LR frames, aligning each supporting frame to the reference before HR reconstruction.TDAN and the SR reconstruction network form the two main sub-networks.
- 3.2. Temporally Deformable Alignment Network: Feature-level alignment avoids explicit motion estimation and image wrapping, while adaptively sampling neighboring features to capture motion cues and image structures.Fractional sampling positions are implemented with bilinear interpolation.
- 3.2. Temporally Deformable Alignment Network: Because implicit alignment is difficult to learn without supervision, TDAN reconstructs aligned LR frames and uses an alignment loss toward the reference frame.The aligned-frame reconstruction is obtained from the aligned feature map with a 3 × 3 convolutional layer.
- 3.3. SR Reconstruction Network: The reconstruction network fuses aligned frames and the reference, maps fused features nonlinearly, and upsamples them to produce the HR frame.Its modules are temporal fusion, nonlinear mapping, and HR frame reconstruction using sub-pixel convolution.
- 3.4. Loss Function: Alignment and SR losses are simultaneously optimized, making the TDAN-based VSR network end-to-end trainable without aligned-frame annotations.The paper describes this training as self-supervised for TDAN.
- 3.5. Analyses of the Proposed TDAN: Compared with optical-flow wrapping, TDAN can explore multiple sampled features around image structures and is presented as a generic replacement for motion compensation.The paper identifies possible applications in video denoising, deblocking, deblurring, frame interpolation, and prediction.
4. Experiments
Experiments evaluate TDAN on benchmark and real-world videos using quantitative, visual, ablation, convergence, and model-size comparisons. TDAN achieves strong VSR performance across tested settings, while a failure case shows that RCAN can recover some structures TDAN misses.
- Experimental Settings: The experiments use Vimeo Super-Resolution for training, Temple frames for validation, and Vid4 sequences for evaluation with PSNR and SSIM metrics.Real-world sequences bldg and CV Book are also evaluated, although their HR frames and degradation methods are unavailable.
- Quantitative Comparisons: On the BI configuration, TDAN achieves the best performance among compared state-of-the-art flow-based VSR and SISR networks.The comparison uses the same training dataset as TOFlow, while other methods are assessed from their provided results.
- Visual Comparisons: Visual comparisons show TDAN restoring finer structures and details than competing methods on BI and BD sequences.Examples include building structures, a window, a baby face, clothing stripes, and cars.
- Quantitative Comparisons: On the BD configuration, TDAN outperforms SPMC and FRVSR, but DUF has higher SSIM while TDAN has higher PSNR and uses fewer input frames.DUF uses 7 frames and FRVSR uses 10, whereas TDAN uses 5 frames; DUF also does not restore frame borders well.
- Model Size: TDAN has fewer parameters than RDN, RCAN, and TOFlow while remaining comparable in size to FRVSR and DUF.Despite its lighter model size, TDAN achieves promising VSR performance.
- Ablation Study: Ablations show MFSR outperforming SISR, TDAN variants outperforming MFSR, and more deformable layers improving TDAN performance; D4 is selected for comparable model size.The convergence study compares SISR, MFSR, and TDAN models with 2–5 deformable convolutional layers.
- Real-World Examples: On real-world sequences with unknown degradation, TDAN produces sharper edges and more image details than the compared state-of-the-art VSR networks.The comparisons use TOFlow under BI degradation and DUF under BD degradation.
- Failure Case: In a city-video failure case, RCAN trained on DIV2K recovers the shown structures accurately, whereas TOFlow and TDAN fail.This comparison identifies a concrete setting where a very deep SISR network outperforms both evaluated VSR methods.
5. Limitation and Failure Exploration
The study identifies dataset resolution, temporal fusion, alignment supervision, and training-label noise as boundaries affecting TDAN’s performance and interpretation.
- Dataset limitations: 448 × 256 HR training frames make it difficult to train very deep networks for recovering finer image structures and details.The authors argue that larger publicly available VSR datasets with 1080P, 2K, or 4K frames would support deeper architectures.
- Failure cases: TDAN fails to recover building structures that RCAN reconstructs accurately, indicating a failure case rather than a universal need for supporting frames.The LR reference frame can provide enough cues for restoring these structures without additional supporting-frame information.
- Fusion design: The temporal fusion module simply concatenates the LR reference and aligned frames before a convolutional layer, leaving room for a more advanced fusion design.The authors state that improved fusion may further improve TDAN-based VSR performance.
- Alignment supervision: An end-to-end variant without explicit aligned-frame prediction or Lalign performs worse than the MFSR baseline, supporting the importance of the alignment loss.The authors attribute this to the relative ease of learning motion from features produced by strong pretrained backbones.
- Label assumptions: Using the LR reference frame as the Lalign label introduces noise because it is not exactly the same as a real aligned LR frame.The authors suggest robust learning methods for label noise as a possible improvement.
6. Conclusion
TDAN is a one-stage feature-level alignment network for VSR that avoids optical-flow estimation and image-wise wrapping. The authors report that its VSR framework outperforms compared flow-based state-of-the-art networks and identify video restoration as a future application area.
- Conclusion: TDAN performs one-stage temporal alignment by implicitly capturing motion with deformable sampling and directly predicting aligned LR frames from sampled features.It avoids the separate motion-estimation and image-wrapping stages used by previous optical-flow-based methods.
- Conclusion: The TDAN-based VSR framework outperforms the compared flow-based state-of-the-art VSR networks.The conclusion attributes this to the one-stage alignment design and contextual information exploration capability.
- Conclusion: The authors propose extending TDAN to video denoising, video deblurring, and video frame interpolation.These are stated as future applications of the proposed alignment network.