Source-linked AI summary
Deep Plug-and-Play Super-Resolution for Arbitrary Blur Kernels
Kai Zhang, Wangmeng Zuo, Lei Zhang
TL;DR
DNN-based SISR is mainly designed for bicubic degradation, leaving arbitrary-blur LR super-resolution difficult. The paper introduces a degradation model compatible with blind kernel estimation and a variable-splitting DPSR framework using super-resolver priors; experiments demonstrate flexibility and effectiveness on blurry LR images.
Problem
Existing DNN-based SISR methods mainly assume bicubic degradation, while super-resolving LR images with arbitrary blur kernels remains challenging.
Method
The paper combines a bicubic-downsampled, blurred, and noisy degradation model with variable-splitting plug-and-play optimization that inserts DNN super-resolvers.
Results
Quantitative and qualitative evaluations on synthetic and real LR images demonstrate that DPSR is flexible and effective for blurry LR images.
Takeaways & Limitations
Plug-and-play SISR can use super-resolver priors rather than being limited to Gaussian denoisers, enabling treatment of arbitrary blur kernels.
Takeaways & Limitations
The work focuses on non-blind SISR for arbitrary uniform blur kernels rather than blind SISR for arbitrary non-uniform blur kernels.
Abstract
from arXiv · showhide
While deep neural networks (DNN) based single image super-resolution (SISR) methods are rapidly gaining popularity, they are mainly designed for the widely-used bicubic degradation, and there still remains the fundamental challenge for them to super-resolve low-resolution (LR) image with arbitrary blur kernels. In the meanwhile, plug-and-play image restoration has been recognized with high flexibility due to its modular structure for easy plug-in of denoiser priors. In this paper, we propose a principled formulation and framework by extending bicubic degradation based deep SISR with the help of plug-and-play framework to handle LR images with arbitrary blur kernels. Specifically, we design a new SISR degradation model so as to take advantage of existing blind deblurring methods for blur kernel estimation. To optimize the new degradation induced energy function, we then derive a plug-and-play algorithm via variable splitting technique, which allows us to plug any super-resolver prior rather than the denoiser prior as a modular part. Quantitative and qualitative evaluations on synthetic and real LR images demonstrate that the proposed deep plug-and-play super-resolution framework is flexible and effective to deal with blurry LR images.
1. Introduction
The paper addresses the mismatch between bicubic-trained SISR methods and realistic LR images by introducing a degradation model and DPSR framework for arbitrary blur kernels. The framework combines blind kernel estimation, variable-splitting optimization, and DNN-based super-resolver priors.
- Motivation: Accurate degradation modeling is crucial for SISR, yet most existing methods assume a fixed degradation model, especially bicubic downsampling.Bicubic degradation supports DNN development but can produce poor results in practical scenarios.
- New degradation model: The proposed degradation model assumes LR images are bicubically downsampled, blurred, and noisy versions of HR images.It generalizes bicubic degradation and enables available blind deblurring methods to estimate blur kernels.
- DPSR framework: DPSR extends plug-and-play SISR by integrating a DNN-based super-resolver into a variable-splitting iterative optimization scheme.The framework replaces the usual Gaussian-denoiser plug-in with a super-resolver prior requiring only a small modification.
- Scope: The paper focuses on non-blind SISR with arbitrary uniform blur kernels rather than blind SISR with arbitrary non-uniform kernels.The authors describe this as an intermediate step toward ultimate blind SISR.
- Contributions: Four contributions are emphasized: a realistic arbitrary-kernel degradation model, a flexible DPSR solver, principled energy minimization, and super-resolver rather than denoiser priors.The method explicitly handles blur distortion in the Fourier domain, supporting arbitrary blur kernels.
2. Related work
Prior DNN-based SISR methods largely target bicubic degradation and often fail on real images or complex blur. Plug-and-play restoration offers a flexible foundation, which this work extends with super-resolver priors.
- Bicubic degradation: DNN-based SISR methods have achieved strong results under bicubic degradation but perform poorly on many real images because of degradation-model mismatch.Methods such as VDSR, FSRCNN, and ESPCN were developed around bicubic inputs.
- Beyond bicubic degradation: Beyond bicubic degradation, existing methods struggle with blur-kernel estimation and are generally designed for Gaussian-like kernels.SRMD considers Gaussian blur, while ZSSR is less effective for severely blurred LR images.
- Plug-and-play restoration: Plug-and-play restoration unrolls an energy function through variable splitting and replaces a prior subproblem with an off-the-shelf Gaussian denoiser.The framework has been applied to problems including denoising, deblurring, super-resolution, and inpainting.
- Plug-and-play restoration: This paper argues that plug-and-play SISR priors need not be limited to Gaussian denoisers and can instead use a simple super-resolver prior.The proposed extension targets more complex SISR degradation while leveraging DNN super-resolvers.
3. Method
DPSR combines a bicubic-downsampled, blurred, noisy degradation model with variable-splitting optimization and a plug-in deep super-resolver. The framework separates blur correction from super-resolution while retaining compatibility with bicubic-trained networks.
- 3.1. New degradation model: The degradation model treats the LR image as a bicubicly downsampled, blurred, and noisy version of a clean HR image.This formulation generalizes bicubic degradation and enables blind deblurring methods to estimate the blur kernel.
- 3.1. New degradation model: Existing deblurring methods can estimate the blur kernel because bicubic downsampling is assumed to produce a clean image before blur and noise are added.The resulting problem is deblurring followed by bicubic SISR.
- 3.2. Deep plug-and-play SISR: The MAP energy combines a degradation-based data-fidelity term with a prior term, explaining why bicubic-trained DNN SISR models perform poorly on real images.The data term uses the proposed degradation model, while the prior represents image regularization.
- 3.2. Deep plug-and-play SISR: Variable splitting introduces an auxiliary variable z, and HQS converts the optimization into alternating subproblems for blur correction and super-resolution.The blur-kernel subproblem pulls the estimate toward a less blurry image, while the super-resolution subproblem maps it toward a clean HR image.
- 3.3. Deep super-resolver prior: The super-resolution subproblem can be replaced by a DNN super-resolver that accepts a noise-level map, allowing existing bicubic-trained models or SRMD to serve as priors.The prior term is implicitly defined by the plugged-in super-resolver.
- 3.3. Deep super-resolver prior: SRResNet+ modifies SRResNet by adding a noise-level input, increasing feature maps from 64 to 96, and removing batch normalization.On Set5, SRResNet+ outperformed SRResNet by an average PSNR gain of 0.15dB under bicubic degradation.
- 3.4. Comparison with related methods: DPSR iteratively optimizes the joint energy instead of cascading deblurring and SISR, avoiding the latter approach’s perturbation-error amplification.The framework is described as well-principled, structurally simple, interpretable, and involving less training.
4. Experiments
Experiments evaluate DPSR on synthetic and real LR images across blur types, noise levels, degradation settings, and super-resolver priors. DPSR achieves strong visual and quantitative performance, converges quickly, and handles estimated kernels across scale factors.
- 4.1. Synthetic LR images: Synthetic evaluations cover Gaussian, motion, and disk blur kernels, with kernel sizes from 5×5 to 35×35 and additional Gaussian noise levels of 2.55 and 7.65.The experiments use synthetic data with ground-truth for quantitative analysis and fair comparison.
- 4.1. Synthetic LR images: DPSR achieves the best PSNR and SSIM performance across the evaluated complex degradation settings on color BSD68.IRCNN+RCAN improves substantially after deblurring, while DeblurGAN+RCAN, GFN, and ZSSR perform poorly under large complex blur.
- 4.1. Synthetic LR images: DPSR consistently produces the best visual results for Gaussian, motion, and disk blur, whereas competing methods show artifacts, residual blur, or degradation mismatch.For motion blur, GFN is inferior to IRCNN+RCAN and DPSR; for Gaussian blur, IRCNN+RCAN produces noise-like artifacts.
- 4.1. Synthetic LR images: DPSR takes about 1.8 seconds per LR image on a single GPU for degradations requiring iterative optimization.For bicubic degradation, iterative solution is not necessarily required because the adopted super-resolver prior is trained end-to-end for that degradation.
- 4.2. LR images with estimated kernel: With estimated blur kernels, DPSR produces the most visually pleasant results on “chip,” “frog,” and “colour,” and one estimated kernel supports different scale factors.The “frog” image is assumed to have no blur; unlike SRMD and ZSSR, DPSR does not require a separate kernel estimate for each scale factor.
5. Conclusion
The paper proposes a deep plug-and-play super-resolution method for LR images with arbitrary blur kernels, combining a new degradation model with variable-splitting optimization.
- The proposed degradation model supports existing blind deblurring methods for blur-kernel estimation.
- Half quadratic splitting integrates a super-resolver prior into the plug-and-play framework instead of a denoiser prior.
- A specialized modular component explicitly handles blur distortion, enabling DPSR to deal with arbitrary blur kernels.
- Extensive experiments demonstrate the feasibility of the degradation model and the effectiveness of DPSR for blurry LR images.