Source-linked AI summary
Deformable 3D Convolution for Video Super-Resolution
Xinyi Ying, Longguang Wang, Yingqian Wang, Weidong Sheng, Wei An, Yulan Guo
TL;DR
Video SR methods often separate spatial feature extraction from temporal motion compensation, limiting joint use of spatio-temporal information. D3Dnet combines deformable and 3D convolution for adaptive joint modeling, and experiments report state-of-the-art SR performance with improved temporal consistency and computational efficiency.
Problem
Existing video SR methods commonly perform spatial feature extraction and temporal motion compensation sequentially, limiting full use of spatio-temporal information.
Method
D3Dnet integrates deformable convolution with 3D convolution so D3D can jointly model appearance and motion with adaptive spatial deformation.
Results
D3Dnet achieves the highest PSNR and SSIM among compared methods, improves temporal-consistency metrics, and uses 16.5% of RCAN’s parameters and 44.5% of its FLOPs.
Takeaways & Limitations
D3Dnet effectively exploits additional temporal information for video SR while maintaining high computational efficiency.
Abstract
from arXiv · showhide
The spatio-temporal information among video sequences is significant for video super-resolution (SR). However, the spatio-temporal information cannot be fully used by existing video SR methods since spatial feature extraction and temporal motion compensation are usually performed sequentially. In this paper, we propose a deformable 3D convolution network (D3Dnet) to incorporate spatio-temporal information from both spatial and temporal dimensions for video SR. Specifically, we introduce deformable 3D convolution (D3D) to integrate deformable convolution with 3D convolution, obtaining both superior spatio-temporal modeling capability and motion-aware modeling flexibility. Extensive experiments have demonstrated the effectiveness of D3D in exploiting spatio-temporal information. Comparative results show that our network achieves state-of-the-art SR performance. Code is available at: https://github.com/XinyiYing/D3Dnet.
I. INTRODUCTION
D3Dnet addresses the limited spatio-temporal modeling of sequential video SR pipelines by combining deformable and 3D convolution. The resulting network is reported to achieve state-of-the-art SR performance with high computational efficiency.
- Sequential feature extraction and motion compensation limit existing video SR methods’ use of spatio-temporal information.
- D3D integrates deformable convolution with C3D to jointly model appearance and motion while retaining spatially adaptive motion modeling.The method deforms only spatial dimensions to incorporate temporal priors and reduce computational cost.
- D3Dnet uses residual deformable 3D convolution blocks to exploit spatio-temporal information for video SR.
- D3Dnet achieves state-of-the-art SR performance with high computational efficiency.
A. Deformable 3D Convolution
D3D extends regular C3D sampling with learnable spatial offsets, producing an adaptive sampling grid for output-feature computation.
- Regular C3D samples a fixed 3×3×3 grid and computes each output feature through weighted summation of sampled input values.The grid contains N = 27 sampling positions.
- D3D uses learnable offsets to enlarge C3D’s spatial receptive field and improve appearance and motion modeling.Offsets are generated from C3D features with 2N channels for height- and width-direction deformations.
- D3D applies learned offsets to deform the plain C3D grid before using the resulting grid to produce the output feature.Fractional offsets are evaluated with bilinear interpolation.
B. Overall Framework
D3Dnet processes seven-frame inputs through C3D, residual D3D blocks, feature fusion, and reconstruction layers to generate super-resolved frames.
- A seven-frame sequence passes through C3D and five residual D3D blocks for motion-aware deep spatio-temporal feature extraction.
- The residual D3D block is designed for simultaneous appearance and motion modeling, while the reconstruction block produces the SR output.
- A bottleneck fuses extracted features, followed by six cascaded residual blocks and a sub-pixel layer for SR reconstruction.
A. Implementation Details
The model is trained on bicubically downsampled Vimeo-90k patches and evaluated on three datasets using spatial-quality and temporal-consistency metrics.
- Training uses Vimeo-90k videos downsampled by 4×, with 32 × 32 LR patches and corresponding HR crops augmented by flipping and rotation.
- Evaluation uses Vid4, Vimeo-90k, and SPMC with PSNR and SSIM for SR quality and MOVIE and T-MOVIE for temporal consistency.All metrics are computed in the luminance channel.
- Optimization uses Adam with an initial learning rate of 4 × 10^-4, halved every six epochs, and training stops after 35 epochs.
1) One-stage vs. Two-stage:
The one-stage variant integrates feature extraction and motion compensation with C3D, while the two-stage variant performs them sequentially. Adding deformable sampling further improves performance over C3D.
- One-stage vs. Two-stage:: The two-stage variant has lower PSNR and SSIM than the one-stage C3D variant.
- One-stage vs. Two-stage:: The two-stage variant sequentially performs spatial feature extraction and deformable alignment, whereas the one-stage variant integrates both with residual C3D blocks.
- One-stage vs. Two-stage:: 0.40 dB PSNR and 0.017 SSIM improve with five resD3D blocks over five resC3D blocks.Each resD3D block adds 0.19M parameters through its offset-generation branch.
3) Context Length:
Using more input frames improves D3Dnet’s quantitative performance, while qualitative results show sharper details and smoother temporal profiles. The evaluations span multiple datasets and report PSNR/SSIM alongside temporal consistency.
- Context Length:: PSNR/SSIM improves from 26.22/0.786 to 26.52/0.799 when input frames increase from 3 to 7.The paper attributes this gain to additional temporal information.
- Context Length:: D3Dnet recovers finer details and produces clearer, smoother temporal profiles than other methods.
- Context Length:: The comparison covers Vid4, Vimeo-90k, and SPMC-11 using PSNR/SSIM results across the evaluated methods.
- Context Length:: Temporal consistency and computational efficiency are additionally evaluated on Vid4.
D. Computational Efficiency
D3Dnet improves SR performance over single-image methods while using fewer parameters and FLOPs than RCAN, and it outperforms compared video methods with a reasonable computational increase.
- D. Computational Efficiency: Computational efficiency is evaluated using parameter count, FLOPs, and running time on Vid4, averaged over 20 runs.
- D. Computational Efficiency: D3Dnet uses 16.5% of RCAN’s parameters and 44.5% of its FLOPs while improving SR performance.
- D. Computational Efficiency: Against video SR methods, D3Dnet achieves better SR performance with a reasonable increase in computational cost.
IV. CONCLUSION
The paper proposes D3Dnet with deformable 3D convolutions to exploit spatio-temporal information for video SR. Experiments report effective temporal-information use and state-of-the-art SR performance.
- IV. CONCLUSION: D3Dnet uses deformable 3D convolutions to model appearance and motion simultaneously for video SR.
- IV. CONCLUSION: Experimental results demonstrate effective use of additional temporal information and state-of-the-art SR performance.