Source-linked AI summary
Blind Super-Resolution With Iterative Kernel Correction
Jinjin Gu, Hannan Lu, Wangmeng Zuo, Chao Dong
TL;DR
Blind SR must recover high-resolution images when real-world blur kernels are complicated and unknown, challenging methods built for predefined kernels. The paper introduces IKC, which iteratively corrects kernel estimates using mismatch artifacts, and SFTMD, an SFT-based multi-kernel SR network. Their combination achieves state-of-the-art blind SR performance on selected kernels and real images, under the paper's stated kernel and noise assumptions.
Problem
Real-world blur kernels are complicated and unavailable, while many SR methods assume predefined kernels, causing severe performance drops under kernel mismatch.
Method
IKC iteratively corrects estimated blur kernels from SR-result artifacts, while SFTMD uses spatial feature transform layers to process multiple blur kernels.
Results
The combination of SFTMD and IKC achieves state-of-the-art performance in blind SR experiments on selected blur kernels and real images.
Takeaways & Limitations
Kernel-mismatch artifacts provide empirical guidance for correcting inaccurate kernels, enabling blind SR with the proposed IKC and SFTMD framework.
Takeaways & Limitations
The formulation focuses on a limited variety of kernels and noise, mainly using isotropic Gaussian blur with bicubic downsampling and Gaussian noise assumptions.
Abstract
from arXiv · showhide
Deep learning based methods have dominated super-resolution (SR) field due to their remarkable performance in terms of effectiveness and efficiency. Most of these methods assume that the blur kernel during downsampling is predefined/known (e.g., bicubic). However, the blur kernels involved in real applications are complicated and unknown, resulting in severe performance drop for the advanced SR methods. In this paper, we propose an Iterative Kernel Correction (IKC) method for blur kernel estimation in blind SR problem, where the blur kernels are unknown. We draw the observation that kernel mismatch could bring regular artifacts (either over-sharpening or over-smoothing), which can be applied to correct inaccurate blur kernels. Thus we introduce an iterative correction scheme -- IKC that achieves better results than direct kernel estimation. We further propose an effective SR network architecture using spatial feature transform (SFT) layers to handle multiple blur kernels, named SFTMD. Extensive experiments on synthetic and real-world images show that the proposed IKC method with SFTMD can provide visually favorable SR results and the state-of-the-art performance in blind SR problem.
1. Introduction
Blind SR addresses the mismatch between known-kernel assumptions in advanced SR methods and complicated, unavailable real-world blur kernels. The paper proposes IKC to iteratively correct kernels and SFTMD to process multiple kernels, achieving state-of-the-art blind SR performance.
- Existing SR methods generally assume a known, predefined blur kernel, whereas real applications involve complicated and unavailable kernels.
- Kernel mismatch causes severe performance drops and regular artifacts, including over-smoothing, over-sharpening, and ringing effects.
- IKC uses the asymmetric artifact patterns caused by kernel mismatch to iteratively correct inaccurate blur-kernel estimates.
- SFTMD uses spatial feature transform layers instead of simple image-kernel concatenation to handle multiple blur kernels.
- The combination of SFTMD and IKC achieves state-of-the-art performance on blind SR across experiments on selected blur kernels and real images.
2. Related Work
Related work spans CNN-based SISR architectures and blind SR methods that estimate degradation kernels or exploit image self-similarity. Deep learning has advanced SISR substantially, but blind SR has received comparatively limited attention.
- CNN-based SISR research has developed many architectures, with residual designs prominent among high-performance networks.
- Blind SR methods treat degradation kernels as unavailable and include approaches based on image self-similarity, unsupervised degradation learning, and internal image recurrence.
- Deep learning methods have dominated SISR, while progress on blind SR has remained limited.
3. Method
The method addresses blind SR by iteratively correcting estimated blur kernels from intermediate SR artifacts and using SFTMD to condition SR processing on kernel information throughout the network.
- Problem: Blind SR is difficult because unknown blur kernels make kernel mismatch produce severe artifacts and amplify estimation errors in SR outputs.The paper focuses mainly on isotropic Gaussian blur with bicubic downsampling and assumes additive Gaussian noise for real-world applications.
- Iterative Kernel Correction: The corrector uses the previous SR image and estimated kernel to update the kernel iteratively, reducing over-smoothing or ringing until the SR result converges.The framework applies small correction steps because one correction may under-correct or over-correct the kernel.
- Iterative Kernel Correction: The predictor estimates an initial low-dimensional kernel representation from the LR image, which produces the first SR result.Kernels are projected into a PCA-reduced space before estimation; the initial estimate is h0 = P(ILR).
- Overall Framework: The proposed IKC framework combines an SR model, predictor, and corrector, while SFTMD extends the SR model to multiple kernels.The final IKC output is the SR result after t correction iterations.
- SFTMD: SFTMD handles multiple blur kernels by applying kernel-conditioned affine transformations to feature maps in middle network layers.SFT layers use scaling and shifting parameters derived from kernel maps, rather than concatenating kernel maps with the image only at the input.
4. Experiments
Experiments evaluate SFTMD and IKC on synthetic and real images, showing improved handling of varying or unknown blur kernels and strong blind-SR results.
- Experiments of SFTMD: SFTMD achieves significantly better performance than SRCNN-CAB and SRMD across Gaussian-kernel settings and datasets.It also outperforms two SRResNet baselines using alternative kernel-concatenation strategies.
- Experiments of SFTMD: Directly concatenating kernel maps with SRResNet feature maps causes severe performance degradation and artifacts.The authors attribute this to interference between direct concatenation and the residual structure.
- Synthetic Test Images: Iterative kernel correction gradually alleviates blurry and ringing artifacts, with corrected kernels producing good visual quality by the 4th iteration.The initial predictor output is unsatisfactory, while increasing iterations improve the SR results.
- Synthetic Test Images: Previous SR methods lose performance as kernel width increases, whereas IKC achieves superior performance under all tested blur kernels.The comparison uses PSNR on BSD100 with SR factor 3 and varying kernel widths.
- Synthetic Test Images: IKC maintains performance under a downsampling setting inconsistent with training, indicating generalization beyond the training degradation process.PCA-based kernel features help IKC learn relationships with SR images rather than only Gaussian kernel width.
- Real Images: On real images without ground-truth HR references, IKC produces artifact-free, sharp results, while representative non-blind and blind methods are less satisfactory.On the Chip image, IKC is visually pleasing and sharp, although SRMD with a hand-crafted kernel has higher contrast and sharper-looking edges.
5. Discussion
The discussion positions IKC as an iterative blind-SR method alongside SFTMD, while noting that experiments mainly use isotropic kernels and future work should address motion blur.
- Discussion: IKC relies on asymmetric mismatch artifacts that can also be observed directionally for slightly anisotropic motion blur.The authors suggest applying directional correction as future work for more realistic motion-blur cases.
- Discussion: The experiments mainly cover isotropic kernels, which the authors note may not apply to some real-world images affected by slight motion blur.This defines the paper’s principal scope boundary.