Source-linked AI summary
Unfolding the Alternating Optimization for Blind Super Resolution
Zhengxiong Luo, Yan Huang, Shang Li, Liang Wang, Tieniu Tan
TL;DR
Blind SR must recover an image while the blur kernel is unknown, and sequentially trained estimation and restoration models can be incompatible. DAN alternates convolutional Estimator and Restorer modules and unfolds their interaction into one end-to-end network. Experiments on synthetic and real-world images report stronger visual results and higher speed than state-of-the-art methods.
Problem
Blind SR is difficult because the blur kernel is unknown, while sequentially trained estimation and restoration models can be incompatible and sensitive to estimation errors.
Method
DAN alternates convolutional Estimator and Restorer modules, unfolding their repeated interaction into an end-to-end trainable network.
Results
Experiments on synthetic datasets and real-world images report that DAN outperforms state-of-the-art methods while producing more visually favorable results at higher speed.
Takeaways & Limitations
The unified alternating design makes kernel estimation use both LR and SR information and trains restoration with estimated rather than ground-truth kernels.
Takeaways & Limitations
The modules must keep their outputs closely related to their conditional inputs; otherwise, iterative results can collapse to a fixed point at the first iteration.
Abstract
from arXiv · showhide
Previous methods decompose blind super resolution (SR) problem into two sequential steps: \textit{i}) estimating blur kernel from given low-resolution (LR) image and \textit{ii}) restoring SR image based on estimated kernel. This two-step solution involves two independently trained models, which may not be well compatible with each other. Small estimation error of the first step could cause severe performance drop of the second one. While on the other hand, the first step can only utilize limited information from LR image, which makes it difficult to predict highly accurate blur kernel. Towards these issues, instead of considering these two steps separately, we adopt an alternating optimization algorithm, which can estimate blur kernel and restore SR image in a single model. Specifically, we design two convolutional neural modules, namely \textit{Restorer} and \textit{Estimator}. \textit{Restorer} restores SR image based on predicted kernel, and \textit{Estimator} estimates blur kernel with the help of restored SR image. We alternate these two modules repeatedly and unfold this process to form an end-to-end trainable network. In this way, \textit{Estimator} utilizes information from both LR and SR images, which makes the estimation of blur kernel easier. More importantly, \textit{Restorer} is trained with the kernel estimated by \textit{Estimator}, instead of ground-truth kernel, thus \textit{Restorer} could be more tolerant to the estimation error of \textit{Estimator}. Extensive experiments on synthetic datasets and real-world images show that our model can largely outperform state-of-the-art methods and produce more visually favorable results at much higher speed. The source code is available at https://github.com/greatlog/DAN.git.
1 Introduction
Blind SR is difficult because real-world blur kernels are unknown, while sequential kernel estimation and image restoration can be incompatible. DAN addresses this by alternately estimating kernels and restoring SR images within one end-to-end network.
- Bicubic-trained SR networks can suffer severe performance drops on real images because their assumed kernels differ from complicated real-world blur kernels.
- Blind SR is challenging because the blur kernel is an undetermined variable and the inverse reconstruction problem is ill-posed.
- Sequential blind-SR methods use independently trained kernel-estimation and restoration models, so small kernel errors can severely degrade the subsequent restoration.
- DAN alternates Restorer and Estimator modules, allowing kernel estimation and SR restoration to occur within a single model.
- The unfolded network is end-to-end trainable, avoids pre/post-processing, and is reported as easier to train and faster than previous two-step solutions.
- Extensive experiments on synthetic datasets and real-world images report better visual results and higher speed than state-of-the-art methods.
2 Related Work
Prior work includes bicubic-specific SR, multi-degradation models conditioned on blur kernels, and sequential blind-SR pipelines combining kernel estimation with non-blind restoration.
- Many SISR methods assume bicubic interpolation, enabling synthetic training pairs but creating a gap from real-image degradations.
- Multi-degradation SR methods accept both an LR image and its corresponding blur kernel, but their outputs can be sensitive to kernel deviations.
- Blind-SR systems commonly combine a kernel-estimation method with a non-blind SR method in a sequential pipeline.
- Some kernel-estimation approaches require two or three separately trained networks, whereas DAN is end-to-end trainable and faster to train.
3 End-to-End Blind Super Resolution
The paper replaces blind SR’s difficult sequential kernel-estimation and image-restoration pipeline with an alternating, end-to-end network whose Estimator and Restorer repeatedly refine each other.
- Problem Formulation: Blind SR jointly determines the blur kernel k and HR image x, making direct optimization difficult because the HR-image prior is usually unknown.Previous methods therefore decompose the problem into sequential kernel estimation and SR restoration.
- Problem Formulation: Sequential methods train separate kernel-estimation and non-blind SR models, creating compatibility, information, and train-test kernel-mismatch drawbacks.The estimator uses only LR information, while the restorer is trained with ground-truth kernels but tested with estimated ones.
- Problem Formulation: DAN alternates convolutional Estimator and Restorer modules, then unfolds a fixed number of iterations into one end-to-end trainable network.The alternating process restores an SR image and estimates its corresponding blur kernel repeatedly; the iteration count is fixed at T.
- Module Interaction: Estimator uses LR and restored SR images, while Restorer uses the LR image and Estimator’s predicted blur kernel, allowing both modules to exchange updated conditional inputs.Restorer is trained with estimated rather than ground-truth kernels, supporting tolerance to estimation error during testing.
- Module Interaction: The conditional residual block concatenates basic and conditional inputs before residual processing, keeping each module’s output related to its changing conditional input.Its residual mapping uses two 3 × 3 convolutions and a channel-attention layer; both modules are built from CRBs.
- Module Architecture: Estimator downsamples the SR image and globally pools features to predict PCA coefficients of the blur kernel, while Restorer stretches the kernel and upsamples features with PixelShuffle.The practical designs use 5 CRBs in Estimator and 40 CRBs in Restorer.
4 Experiments
Experiments evaluate DAN on isotropic and irregular blur settings using synthetic training and benchmark data, showing strong quantitative, visual, kernel-estimation, convergence, and speed results.
- Data, Training and Testing: The experiments use 3,450 DIV2K and Flickr2K HR images with isotropic Gaussian and more general irregular blur-kernel settings.Testing uses standard benchmarks including Set5, Set14, Urban100, BSD100, Manga109, and DIV2KRK.
- Quantitative Results: In Setting 1, DAN largely outperforms IKC across scales and datasets, including a 4.95dB PSNR advantage on Manga109 at scale ×3.The comparison includes ZSSR, IKC, and CARN-based pipelines.
- Visual Results: Visual comparisons show DAN restoring sharper edges in Urban100 and cleaner, more reliable details in DIV2KRK than competing methods.CARN and ZSSR fail to restore some window edges, while KernelGAN plus ZSSR produces artifacts on DIV2KRK.
- Quantitative Results: In Setting 2, DAN outperforms KernelGAN plus ZSSR by 2.20dB at scale ×2 and 0.74dB at scale ×4.The irregular-kernel setting is described as more general and more difficult, while bicubic-trained methods and NTIRE methods generalize poorly.
- Study of Estimated Kernels: DAN predicts kernels with lower reduced-space L1 error than IKC, and its performance gap grows as sigma increases.The authors interpret these results as suggesting more accurate kernel retrieval and better generalization ability.
- Iterations and Inference Speed: Testing beyond the four training iterations shows gradual convergence without performance degradation, while DAN reaches 0.75 seconds per image—5 times faster than IKC.It is nearly 554 times faster than KernelGAN plus ZSSR, and the iteration study suggests robustness to intermediate estimation error.
5 Conclusion
The paper concludes that DAN is an end-to-end blind SR algorithm built by unfolding alternating optimization between Restorer and Estimator. This design improves kernel estimation, increases Restorer tolerance to estimation errors, and achieves large gains over state-of-the-art blind SR methods.
- DAN unfolds alternating optimization between convolutional Restorer and Estimator modules into an end-to-end trainable network.The alternating process can substantially improve both modules during iterations.
- Estimator uses both LR and SR images, making blur-kernel estimation easier.
- Restorer is trained with Estimator-predicted kernels rather than ground-truth kernels, potentially increasing tolerance to estimation errors.
- DAN outperforms state-of-the-art blind SR methods by a large margin.
Broader Impact
Super Resolution has broad applications including video enhancement, medical imaging, and security and surveillance imaging. The proposed DAN method is described as improving performance and speed for unknown degradations, with possible real-time applications and no foreseeable negative societal consequence.
- Super Resolution supports applications in video enhancement, medical imaging, and security and surveillance imaging.
- DAN could improve these applications when image degradations are unknown.
- DAN has relatively better performance and much higher speed, potentially supporting real-time video enhancement or surveillance imaging.
- The paper reports no foreseeable negative societal consequence from the proposed method.