Deep Unfolding Network for Image Super-Resolution

Kai Zhang; Luc Van Gool; Radu Timofte

Source-linked AI summary

Deep Unfolding Network for Image Super-Resolution

Kai Zhang, Luc Van Gool, Radu Timofte

arXiv:2003.10428v1Mar 23, 2020eess.IVcs.CV

TL;DR

Learning-based SISR is effective but generally lacks model-based flexibility across scale factors, blur kernels, and noise levels. USRNet unfolds MAP inference into an end-to-end trainable network with neural data and prior modules, and it performs favorably across degradation settings with a single model. Its scope is non-blind SISR, where the LR image, blur kernel, and noise level are known beforehand.

Problem
Learning-based SISR generally lacks the flexibility to handle different scale factors, blur kernels, and noise levels that model-based methods support.
Method
USRNet unfolds half-quadratic-splitting MAP inference into alternating data and prior subproblems implemented by neural modules.
Results
USRNet performs favorably on LR images with different degradation settings, and a single model significantly outperforms competing methods across scale factors, blur kernels, and noise levels.
Takeaways & Limitations
USRNet combines model-based degradation and prior constraints with end-to-end learning for flexible practical super-resolution.
Takeaways & Limitations
The method focuses on non-blind SISR, assuming the LR image, blur kernel, and noise level are known beforehand.

Abstract

from arXiv · show

Learning-based single image super-resolution (SISR) methods are continuously showing superior effectiveness and efficiency over traditional model-based methods, largely due to the end-to-end training. However, different from model-based methods that can handle the SISR problem with different scale factors, blur kernels and noise levels under a unified MAP (maximum a posteriori) framework, learning-based methods generally lack such flexibility. To address this issue, this paper proposes an end-to-end trainable unfolding network which leverages both learning-based methods and model-based methods. Specifically, by unfolding the MAP inference via a half-quadratic splitting algorithm, a fixed number of iterations consisting of alternately solving a data subproblem and a prior subproblem can be obtained. The two subproblems then can be solved with neural modules, resulting in an end-to-end trainable, iterative network. As a result, the proposed network inherits the flexibility of model-based methods to super-resolve blurry, noisy images for different scale factors via a single model, while maintaining the advantages of learning-based methods. Extensive experiments demonstrate the superiority of the proposed deep unfolding network in terms of flexibility, effectiveness and also generalizability.

1. Introduction

SISR seeks to recover sharp HR detail from LR images, but existing learning-based methods lack the flexibility of model-based approaches across degradation settings. USRNet addresses this by unfolding model-based inference into an end-to-end trainable network with explicit degradation and prior constraints.

SISR recovers a natural, sharp HR image from an LR image and supports display enhancement and other high-level vision tasks.
The classical degradation model represents an LR image as a blurred, downsampled, and noisy version of an HR image.The blur kernel, scale factor, and additive AWGN level characterize the degradation.
Existing learning-based SISR methods commonly assume bicubic degradation, limiting their direct applicability to more practical blur and noise settings.Model-based methods accommodate broader degradation models but are harder to evaluate and develop under bicubic assumptions.
USRNet unfolds a half-quadratic-splitting inference into alternating data and prior subproblems solved by neural modules.This produces an end-to-end trainable iterative network that bridges model-based and learning-based approaches.
USRNet imposes degradation consistency and natural-image prior constraints while handling different scale factors, blur kernels, and noise levels with one model.

2. Related work

Related work spans diverse degradation models, flexible plug-and-play methods, and deep unfolding approaches. The paper positions USRNet as an end-to-end alternative that retains flexibility while addressing computational and optimization limitations.

2.1. Degradation models: SISR degradation models include direct downsampling, noisy bicubic degradation, Gaussian blur with downsampling, and blurred bicubic degradation.
2.1. Degradation models: CNN-based SISR has received limited attention for the classical degradation model despite the variety of proposed degradation formulations.
2.2. Flexible SISR methods: Flexible super-resolution should account for scale factor, blur kernel, and noise level rather than relying only on bicubic degradation.
2.2. Flexible SISR methods: Deep plug-and-play methods handle varied degradations by inserting learned CNN priors into MAP iterations, but require high computation and manually selected hyperparameters.
2.2. Flexible SISR methods: This work addresses non-blind SISR, assuming the LR image, blur kernel, and noise level are known beforehand.These quantities may be estimated or known from other information, and users can tune sharpness and smoothness through them.
2.3. Deep unfolding methods: Deep unfolding methods optimize parameters end to end and can integrate degradation constraints, but prior, data-subproblem, and training-stage limitations remain.

3. Method

USRNet unfolds MAP super-resolution into alternating data and prior updates, combining degradation-aware model-based inference with end-to-end trainable neural modules. Its data, prior, and hyper-parameter modules support varied blur, scale, noise, and related restoration settings.

3.1. Degradation model: classical vs. bicubic: Bicubic degradation can be approximated by estimating blur kernels within the classical degradation model for scale factors 2, 3, and 4.The estimated kernels have center shifts of 0.5, 1, and 1.5 pixels toward the upper-left, respectively, because downsampling selects upper-left pixels.
3.2. Unfolding optimization: The MAP energy combines a degradation-based data term, an image prior, and a trade-off parameter controlling their balance.HQS introduces an auxiliary variable and penalty term, decoupling the data and prior subproblems for iterative optimization.
3.2. Unfolding optimization: HQS iteratively increases its penalty parameter so the auxiliary and primary variables approach equality without incurring unnecessarily slow convergence.The k-th iteration uses penalty parameter µk, while the data and prior terms are solved separately.
3.3. Deep unfolding network: The data module implements the closed-form FFT solution, enforcing the degradation constraint while estimating a clearer HR image from the observed LR image and current estimate.It accepts the scale factor s and blur kernel k as inputs and uses trade-off hyper-parameter αk; its implementation has no trainable parameters.
3.3. Deep unfolding network: The prior module uses a noise-conditioned ResUNet denoiser, while the hyper-parameter module controls data and prior outputs through positive αk and βk values.USRNet alternates data and prior modules for K iterations; the hyper-parameter module uses three fully connected layers with ReLU and Softplus activations.
3.4. End-to-end training: USRNet is trained end-to-end on synthesized degradations with scale factors {1, 2, 3, 4}, using L1 loss for PSNR and USRGAN fine-tuning for perceptual quality.The training data combine DIV2K and Flickr2K; USRGAN uses weighted L1, VGG perceptual, and relativistic adversarial losses and only considers scale factor 4.

4. Experiments

Experiments show that USRNet handles varied scale factors, blur kernels, and noise levels with strong quantitative performance, while USRGAN and USRNet produce favorable visual results and generalize beyond training settings.

PSNR results: USRNet with a single model significantly outperforms competing methods across different scale factors, blur kernels, and noise levels.It achieves at least a 1dB average PSNR gain over IRCNN with 30 iterations while using much fewer iterations.
PSNR results: USRNet remains favorable on bicubic degradation despite not being trained on bicubic kernels.The evaluation uses approximated bicubic kernels as input.
Visual results: USRGAN produces more visually pleasant results than USRNet, while USRNet recovers sharper edges than IRCNN on motion-blurred images.Both USRNet and IRCNN outperform RCAN and IKC visually for the motion-blur example, but neither produces realistic textures.
Analysis on D and P: Across iterations, the data and prior modules alternately support blur removal and detail recovery, while the prior module enhances high-frequency details.Removing the data module lowers average PSNR by about 0.1dB for Gaussian kernels and 0.3dB for motion kernels.
Analysis on H: The hyper-parameter module adapts α and β to degradation conditions: α increases with noise, while β increases with scale factor and noise level and decreases across iterations.These trends indicate progressively reduced noise in HR estimations and larger β values for more complex degradation.
Generalizability: USRNet and USRGAN generalize to kernels much larger than the 25×25 training size, and USRGAN trained at scale factor 4 also produces visual results at scale factor 3.The reported generalizability is attributed to the trainable parameter-free data module and the prior module's cross-scale behavior.

5. Conclusion

The paper proposes a deep unfolding super-resolution network that combines model-based degradation handling with end-to-end learning. Experiments demonstrate flexibility, effectiveness, and generalizability for varied degraded LR images.

5. Conclusion: USRNet integrates data, prior, and hyper-parameter modules to impose degradation and prior constraints on the super-resolution solution.The network handles the classical degradation model with different settings through a single model.
5. Conclusion: The proposed deep unfolding architecture demonstrates flexibility, effectiveness, and generalizability for super-resolving various degraded LR images.Its design combines the flexibility of model-based methods with the advantages of learning-based methods.