Source-linked AI summary
Spatio-Temporal Filter Adaptive Network for Video Deblurring
Shangchen Zhou, Jiawei Zhang, Jinshan Pan, Haozhe Xie, Wangmeng Zuo, Jimmy Ren
TL;DR
Video deblurring must handle spatially variant blur while existing optical-flow-based alignment and kernel approximation can fail when motion estimates are inaccurate. STFAN unifies alignment and deblurring with dynamically generated FAC filters from recurrent frame inputs. The paper reports favorable accuracy, speed, and model size against state-of-the-art methods, while selecting k = 5 as a trade-off among complexity, model size, and performance.
Problem
Optical-flow-based video deblurring can produce artifacts or ineffective restoration because motion estimation from blurry adjacent frames is challenging.
Method
STFAN uses FAC layers and dynamically generated element-wise filters to perform feature-domain frame alignment and spatially variant deblurring in one framework.
Results
The method performs favorably against state-of-the-art algorithms in accuracy, speed, and model size on benchmark evaluations.
Takeaways & Limitations
The unified filter-adaptive formulation lets the network use features from different time steps without explicit motion estimation and handle spatially variant blur in dynamic scenes.
Takeaways & Limitations
The adaptive filter size is set to k = 5 as a trade-off among computational complexity, model size, and performance.
Abstract
from arXiv · showhide
Video deblurring is a challenging task due to the spatially variant blur caused by camera shake, object motions, and depth variations, etc. Existing methods usually estimate optical flow in the blurry video to align consecutive frames or approximate blur kernels. However, they tend to generate artifacts or cannot effectively remove blur when the estimated optical flow is not accurate. To overcome the limitation of separate optical flow estimation, we propose a Spatio-Temporal Filter Adaptive Network (STFAN) for the alignment and deblurring in a unified framework. The proposed STFAN takes both blurry and restored images of the previous frame as well as blurry image of the current frame as input, and dynamically generates the spatially adaptive filters for the alignment and deblurring. We then propose the new Filter Adaptive Convolutional (FAC) layer to align the deblurred features of the previous frame with the current frame and remove the spatially variant blur from the features of the current frame. Finally, we develop a reconstruction network which takes the fusion of two transformed features to restore the clear frames. Both quantitative and qualitative evaluation results on the benchmark datasets and real-world videos demonstrate that the proposed algorithm performs favorably against state-of-the-art methods in terms of accuracy, speed as well as model size.
1. Introduction
Video deblurring is hindered by spatially variant blur and difficult motion estimation, while STFAN unifies alignment and deblurring through dynamically generated feature filters. The proposed FAC layers support this formulation, and experiments report favorable accuracy, speed, and model size.
- Camera shake and high-speed motion create blurry videos that impair visual quality and high-level tasks such as tracking, stabilization, and SLAM.
- Optical-flow-based methods can align frames or approximate non-uniform blur, but estimating motion from blurry adjacent frames is challenging and inaccurate alignment causes artifacts.
- STFAN dynamically generates alignment and deblurring filters from blurry and restored previous-frame images plus the current blurry image.
- The FAC layer applies generated element-wise filters to feature transformation for both spatially variant alignment and deblurring.
- Quantitative and qualitative evaluations report favorable performance against state-of-the-art algorithms in accuracy, speed, and model size.
2. Related Work
Related work addresses dynamic-scene deblurring through spatially varying kernels, optical flow, multi-image information, and learned kernel prediction. These approaches motivate a unified filter-adaptive treatment of alignment and deblurring.
- Single-Image Deblurring: Single-image methods struggle to model spatially varying blur in dynamic scenes, motivating dense motion fields and pixel-wise blur-kernel estimation.
- The proposed work formulates neighboring-frame alignment and non-uniform blur removal as two element-wise filter-adaptive convolution processes.
- Single-Image Deblurring: CNN-based methods estimate non-uniform blur kernels, but line-shaped kernels can be inaccurate and conventional non-blind deblurring is time-consuming.
- Multi-Image Deblurring: Multi-image methods use optical flow to segment blur layers, approximate blur kernels, or exploit stereo depth, but their approaches depend on estimated motion or scene geometry.
- Kernel Prediction Network: Kernel prediction networks generate input-conditioned filters and apply them through dynamic filtering layers for low-level vision tasks.
3. Proposed Algorithm
STFAN unifies recurrent frame alignment and spatially variant deblurring through dynamically generated, channel-specific FAC filters applied to down-sampled features. Two transformed feature streams are fused for reconstruction, with MSE and perceptual losses guiding training.
- Network Architecture: STFAN uses previous blurry and restored frames plus the current blurry frame to generate adaptive alignment and deblurring filters.The triplet input supports motion capture across frames and implicitly exploits the previous frame's blur kernel.
- Filter Adaptive Convolutional Layer: FAC applies dynamically generated element-wise, channel-specific filters to feature transformation for spatially variant alignment and deblurring.Unlike standard dynamic-filter approaches, FAC operates on down-sampled features and uses within-channel convolution.
- Filter Adaptive Convolutional Layer: Down-sampled feature processing provides a larger receptive field without requiring large filters, reducing computational and memory demands.The paper reports that a small intermediate feature-layer filter, such as size 5, is sufficient for large motions and blurs.
- Network Architecture: Two FAC layers align deblurred previous-frame features and deblur current-frame features before concatenation and reconstruction.The concatenated features are also passed forward through a convolutional layer for recurrent propagation.
- Loss Function: The reconstruction objective combines MSE between restored and sharp frames with perceptual loss based on VGG-19 features.The perceptual term uses conv3-3 features, and its weight λ is set to 0.01.
4. Experiments
Experiments evaluate STFAN on benchmark and real blurry videos, measuring restoration quality, efficiency, and model size. The results show favorable quantitative and qualitative performance, including sharper outputs, temporal consistency, lower runtime, and smaller models.
- Experimental Setup: The dataset contains 71 videos and 6,708 blurry-sharp pairs, split into 61 training videos and 10 testing videos.The training split contains 5,708 pairs, while the testing split contains 1,000 pairs.
- Quantitative Evaluation: Table 1 evaluates video deblurring methods using PSNR, SSIM, running time, and parameter numbers.The comparison includes conventional optical-flow-based and CNN-based video deblurring methods, along with image deblurring methods.
- Quantitative and Qualitative Evaluation: On the dynamic-scene video deblurring test set, STFAN performs favorably against state-of-the-art algorithms.Figure 4 reports that STFAN restores sharper images with more details, while existing methods struggle with sharp details and non-uniform blur.
- Quantitative and Qualitative Evaluation: On real blurry videos, STFAN restores sharper images with more image details and robustly handles unknown real blur in dynamic scenes.Figure 5 qualitatively compares STFAN with state-of-the-art image and video deblurring methods.
- Running Time and Model Size: STFAN has less running time and a smaller model size than existing end-to-end CNN methods.The paper attributes this efficiency to aligned deblurred previous-frame features and the FAC layer; method [10] is slightly faster and smaller but performs worse according to the comparison.
- Temporal Consistency: A recurrent network transfers previous feature maps over time, while FAC explicitly aligns information between consecutive frames to improve temporal consistency.The reported video results show sharper frames with better temporal consistency.
5. Analysis and Discussions
The analysis evaluates STFAN’s adaptive filters, FAC-based alignment and deblurring, branch contributions, input choices, and filter-size trade-offs. Results show that these design components improve feature transformation and restoration, while the triplet input and k = 5 filter size balance performance and efficiency.
- Effectiveness of the FAC layers: The generated alignment filters are coherent with vehicle motion estimated by EpicFlow, while deblurring filters differ between foreground and background regions.The network estimates spatially different filters because removing different blur requires different operations.
- Effectiveness of the FAC layers: FAC alignment correctly warps vehicle features, and FAC deblurring produces sharper transformed features than the pre-FAC features.These observations support alignment and blur removal directly in the feature domain.
- Effectiveness of the FAC layers: Removing one or both FAC layers worsens network performance, and the network cannot restore an equally sharp image without FAC layers.Table 2 compares variants that replace feature transformation with direct feature concatenation.
- Effectiveness of the A and D Branches: Removing either the alignment branch or the deblurring branch produces unsatisfying deblurring results compared with the full network.The comparison isolates the contributions of the A and D branches.
- Input and filter-size analysis: The triplet input (B_t−1, R_t−1, B_t) achieves the best performance among the tested input variants.The authors state that this input helps the network implicitly capture motion and model dynamic-scene blur.
- Input and filter-size analysis: Larger adaptive filters improve performance, but gains after k = 5 are minor; the authors set k = 5 as a trade-off among complexity, model size, and performance.This choice balances computational cost, model size, and restoration quality.
6. Conclusion
The conclusion presents STFAN as a spatio-temporal video deblurring network based on FAC layers. By formulating alignment and deblurring as filter-adaptive processes, it uses features across time without explicit optical-flow estimation and handles spatially variant blur in dynamic scenes.
- 6. Conclusion: STFAN dynamically generates element-wise alignment and deblurring filters and applies them through FAC layers in the feature domain.The two filter-adaptive processes perform temporal alignment and deblurring sequentially.
- 6. Conclusion: The filter-adaptive formulation uses features from different time steps without explicit motion estimation and handles spatially variant blur in dynamic scenes.The conclusion identifies this as the central capability of the proposed design.
- 6. Conclusion: Experiments demonstrate effectiveness in accuracy, speed, and model size.The conclusion reports this overall evaluation outcome without giving specific metric values.