Source-linked AI summary
Iterative Filter Adaptive Network for Single Image Defocus Deblurring
Junyong Lee, Hyeongseok Son, Jaesung Rim, Sunghyun Cho, Seungyong Lee
TL;DR
Single-image defocus deblurring must cope with spatially varying and large blur that challenges fixed blur models. The paper proposes IFAN with iterative separable-filter convolution and disparity-estimation/reblurring training, achieving state-of-the-art performance on real-world images. Its main scope boundary is significantly large or irregular defocus blur.
Problem
Single-image defocus deblurring remains challenging because blur size and shape vary across an image, including significantly large blur.
Method
IFAN predicts per-pixel stacks of separable deblurring filters, applies them iteratively with IAC, and uses disparity estimation and reblurring during training.
Results
The method achieves state-of-the-art deblurring performance, with filter prediction and IAC producing PSNR and LPIPS gains over the baseline in ablation experiments.
Takeaways & Limitations
IFAN provides an end-to-end framework for efficiently handling spatially varying and large defocus blur in real-world images.
Takeaways & Limitations
The network remains limited on significantly large blur and may not properly handle irregular or strong-highlight bokeh.
Abstract
from arXiv · showhide
We propose a novel end-to-end learning-based approach for single image defocus deblurring. The proposed approach is equipped with a novel Iterative Filter Adaptive Network (IFAN) that is specifically designed to handle spatially-varying and large defocus blur. For adaptively handling spatially-varying blur, IFAN predicts pixel-wise deblurring filters, which are applied to defocused features of an input image to generate deblurred features. For effectively managing large blur, IFAN models deblurring filters as stacks of small-sized separable filters. Predicted separable deblurring filters are applied to defocused features using a novel Iterative Adaptive Convolution (IAC) layer. We also propose a training scheme based on defocus disparity estimation and reblurring, which significantly boosts the deblurring quality. We demonstrate that our method achieves state-of-the-art performance both quantitatively and qualitatively on real-world images.
1. Introduction
Single-image defocus deblurring must address blur whose size and shape vary spatially. The paper introduces IFAN and defocus-specific training to improve restoration on real-world images.
- Motivation: Defocus deblurring restores all-in-focus images from defocused inputs, supporting photographers and high-level vision tasks.Restored images can facilitate semantic segmentation and object detection.
- Motivation: Conventional model-based methods often fail because disc or Gaussian blur assumptions restrict the nonlinear blur shapes found in real images.They estimate per-pixel kernels and apply non-blind deconvolution under predefined blur models.
- IFAN: IFAN predicts spatially adaptive per-pixel deblurring filters and applies them to defocused features to handle spatially varying blur.Unlike direct pixel prediction, the network generates filters from the input image and uses them to produce deblurred features.
- IFAN: Stacks of small-sized separable filters and the Iterative Adaptive Convolution layer provide large receptive fields while keeping computational cost low.IAC iteratively applies the predicted separable filters to features, improving deblurring for large blur.
- Training: Defocus disparity estimation and reblurring are introduced as training tasks to improve prediction of blur magnitudes, shapes, and sizes.Dual-pixel disparities are proportional to blur magnitudes, while reblurring trains filters to reproduce the defocused input from an all-in-focus image.
- Evaluation: The method is evaluated on real-world datasets and reports state-of-the-art deblurring accuracy and computational cost.Evaluation includes DPDD, the Pixel dual-pixel test set, CUHK blur detection, and the introduced RealDOF test set.
2. Related Work
Related defocus-deblurring methods estimate blur using specified models and then perform non-blind deconvolution, while filter adaptive networks process features with spatially adaptive filters.
- Defocus deblurring: Most classical and deep-learning defocus-deblurring methods estimate a defocus map from hand-crafted, learned, or deep features under a specific blur model.They then use non-blind deconvolution to restore a sharp image.
- Filter adaptive networks: Filter adaptive networks combine spatially adaptive filter prediction with feature transformation through filter adaptive convolution.This framework has been applied to frame interpolation, denoising, super-resolution, segmentation, and related tasks.
- Filter adaptive networks: Prior adaptive methods include pixel-recurrent convolution for motion deblurring and FAC for frame alignment and video deblurring.Pixel-recurrent processing requires massive computational cost, while prior FAC work motivates the present design.
3. Single Image Defocus Deblurring
The paper introduces IFAN and IAC for single-image defocus deblurring with spatially varying and large blur. It combines adaptive separable filtering with disparity-estimation and reblurring supervision, achieving stronger results on real-world benchmarks.
- 3.1. Iterative Adaptive Convolution (IAC): IAC iteratively applies small-sized separable filters to enlarge receptive fields with little computational overhead.This addresses the memory and computational costs of using large conventional filters.
- 3.2. Deblurring Network with IFAN: IFAN predicts spatially adaptive per-pixel deblurring filters and applies them to defocused features to produce deblurred features.Its IAC layer processes stacks of separable filters channel-wise and iteratively.
- 3.3. Network Training: The network uses dual-pixel disparities for defocus disparity estimation and an auxiliary reblurring network during training.Dual-pixel data are used only during training, while the trained network requires a single defocused image at test time.
- 4.1. Ablation Study: 0.44% PSNR and 16.31% LPIPS gains occur when disparity estimation is added to filter prediction and IAC, exceeding the corresponding baseline gains.The ablation attributes this improvement to explicit deblurring-filter utilization of dual-pixel supervision.
- 4.1. Ablation Study: The model with both disparity estimation and reblurring achieves the best performance in every measure, indicating complementary effects.The two training components are reported to have synergistic effects.
- 4.2. Comparison with Previous Methods: On DPDD and RealDOF, the method outperforms prior approaches, including single-image DPDNetS and dual-pixel DPDNetD, while using a single defocused input at test time.On RealDOF, it exceeds DPDNetS by more than 2 dB in PSNR and generalizes across cameras.
5. Conclusion
The framework combines IFAN, iterative adaptive convolution, and defocus-specific training to handle spatially varying blur, while remaining limited on very large or irregular defocus blur.
- IFAN predicts per-pixel separable deblurring filters to flexibly handle spatially varying defocus blur.
- Iterative Adaptive Convolution applies separable filters iteratively for efficient and effective handling of large blur.
- Defocus disparity estimation and reblurring improve IFAN’s estimation of defocus blur from a single image.
- The method achieves state-of-the-art performance, with experiments verifying the effect of each model component.
- The network remains limited for significantly large blur and may not properly handle irregular or strongly highlighted bokeh.