Deep Generalized Unfolding Networks for Image Restoration

Chong Mou; Qian Wang; Jian Zhang

Source-linked AI summary

Deep Generalized Unfolding Networks for Image Restoration

Chong Mou, Qian Wang, Jian Zhang

arXiv:2204.13348v1Apr 28, 2022cs.CVeess.IV

TL;DR

Image restoration methods often sacrifice interpretability or require known degradation models, limiting their use on complex real-world data. DGUNet unfolds PGD with learned gradient estimation and informative inter-stage pathways, and experiments across twelve test sets report state-of-the-art performance, interpretability, and generalizability.

Problem
Existing deep restoration methods are often black boxes, while hybrid and unfolding methods commonly require predefined degradation processes or handcrafted assumptions for complex real-world applications.
Method
DGUNet unfolds PGD into an end-to-end trainable network with gradient estimation, multi-scale spatial-adaptive inter-stage pathways, and informative proximal mapping.
Results
DGUNet achieves state-of-the-art performance, interpretability, and generalizability across numerous image restoration tasks, including twelve synthetic and real-world test sets.
Takeaways & Limitations
DGUNet combines model-based interpretability with deep-learning adaptability for general image restoration tasks.
Takeaways & Limitations
Traditional PGD with handcrafted ℓ1 regularization has limited representation ability and is restricted to a few degradation-known tasks.

Abstract

from arXiv · show

Deep neural networks (DNN) have achieved great success in image restoration. However, most DNN methods are designed as a black box, lacking transparency and interpretability. Although some methods are proposed to combine traditional optimization algorithms with DNN, they usually demand pre-defined degradation processes or handcrafted assumptions, making it difficult to deal with complex and real-world applications. In this paper, we propose a Deep Generalized Unfolding Network (DGUNet) for image restoration. Concretely, without loss of interpretability, we integrate a gradient estimation strategy into the gradient descent step of the Proximal Gradient Descent (PGD) algorithm, driving it to deal with complex and real-world image degradation. In addition, we design inter-stage information pathways across proximal mapping in different PGD iterations to rectify the intrinsic information loss in most deep unfolding networks (DUN) through a multi-scale and spatial-adaptive way. By integrating the flexible gradient descent and informative proximal mapping, we unfold the iterative PGD algorithm into a trainable DNN. Extensive experiments on various image restoration tasks demonstrate the superiority of our method in terms of state-of-the-art performance, interpretability, and generalizability. The source code is available at https://github.com/MC-E/Deep-Generalized-Unfolding-Networks-for-Image-Restoration.

1. Introduction

Image restoration combines degradation fidelity with priors, but existing approaches trade interpretability, adaptability, or computational efficiency. DGUNet unfolds PGD while adding learned gradient estimation and inter-stage pathways for complex restoration tasks.

Motivation: Image restoration recovers a high-quality image from degraded measurements, commonly modeled with a degradation matrix and additive noise.The problem is typically ill-posed and spans denoising, deblurring, deraining, and compressive sensing.
Motivation: Handcrafted priors such as total variation, sparse representation, low-rankness, and self-similarity have limited representation ability and can yield unstable, time-consuming inference.
Motivation: Deep-learning restoration learns strong priors and enables fast inference, but its black-box design makes component roles and performance gains difficult to analyze.
Motivation: Hybrid plug-and-play and unfolding methods improve interpretability or efficiency, yet most deep unfolding networks require known degradation processes that may be complicated or uncertain in real applications.
Proposed approach: DGUNet unfolds PGD into an end-to-end trainable, interpretable network, combining gradient estimation for degradation-unknown cases with multi-scale, spatial-adaptive inter-stage pathways.
Results: Extensive experiments across twelve synthetic and real-world test sets report state-of-the-art performance and attractive complexity.

2. Related Works

Traditional model-based restoration separates data fidelity and regularization through iterative optimization, while deep methods learn restoration mappings but often obscure algorithmic roles. Deep unfolding implements optimization iterations as trainable recurrent network blocks.

Model-based methods: Model-based restoration formulates image recovery as Bayesian MAP optimization with separate data fidelity and regularization terms.Algorithms such as HQS, ADMM, and PGD alternately solve data and prior subproblems.
Deep learning methods: Deep-learning restoration methods use CNN architectures and learned features to improve low-level image processing performance.
Deep unfolding methods: DGUNet’s architecture is organized into PGD-aligned stages, each containing a flexible gradient descent module and an informative proximal mapping module.
Deep unfolding methods: Deep unfolding represents conventional iterative optimization algorithms as stacks of recurrent DNN blocks trained end-to-end for specific tasks.Earlier plug-and-play methods use trained denoisers to implicitly express the regularization term, while unfolding makes network components trainable.

3. Methodology

DGUNet unfolds Proximal Gradient Descent into a trainable network with flexible gradient descent for known or unknown degradations and informative proximal mapping that preserves inter-stage features.

Network architecture: DGUNet unfolds PGD into repeated neural-network stages, each pairing a flexible gradient descent module with an informative proximal mapping module.The default network uses seven stages with shared parameters, while DGUNet+ makes stages parameter-independent.
Flexible Gradient Descent Module: When the degradation matrix A is known, FGDM uses the accurate gradient with a trainable stage-specific step size ρ^k.The update is v^k = x̂^{k−1} − ρ^k A^T(Ax̂^{k−1} − y).
Flexible Gradient Descent Module: When A is unknown, FGDM predicts the gradient data-drivenly using residual blocks that simulate A and its transpose instead of task-specific degradation assumptions.This design retains the gradient-descent role while extending PGD to degradation-unknown cases.
Informative Proximal Mapping Module: IPMM implements proximal mapping with an hourglass encoder-decoder that extracts multi-scale features using channel attention and residual blocks.The module treats proximal mapping as a denoising problem and uses features at three scales.
Informative Proximal Mapping Module: Inter-stage feature pathways broadcast encoder and decoder features across PGD stages, using spatial-adaptive normalization to preserve spatially refined memory.At each scale, previous-stage features are embedded by independent 1 × 1 convolutions, added, and used to modulate current encoder features.
Loss Function Design: The network is trained with an ℓ2 loss over restoration outputs from all stages, using the degraded measurement y and ground-truth image x.K denotes the total number of stages, and the trainable parameters include the stage step sizes and proximal-mapping parameters.

4. Experiments

DGUNet is evaluated across deraining, deblurring, denoising, and compressive sensing, with comparisons on standard benchmarks and ablations of its main design choices. The method achieves strong restoration results, while inter-stage fusion, spatial adaptation, gradient descent, and stage depth each affect performance.

Image Deraining Results: DGUNet and DGUNet+ outperform competing methods across five deraining test sets, including gains of 1.85 dB on Rain100L and 0.65 dB on Rain100H versus MPRNet.Their average gain over MPRNet across the five test sets is 0.73 dB.
Image Deblurring Results: DGUNet and DGUNet+ outperform competing methods on GoPro and HIDE deblurring tests; DGUNet+ exceeds MPRNet by 0.51 dB and 0.44 dB, respectively.The datasets contain real-scene degradation factors, including camera response and human-aware motion blur.
Image Denoising Results: DGUNet achieves the best performance on both SIDD and DND denoising tests, while DGUNet+ exceeds MPRNet by 0.32 dB on DND and 0.20 dB on SIDD.Visual examples show robustness to high- and low-intensity noise while recovering textures and structures.
Compressive Sensing Results: DGUNet and DGUNet+ have advantages over classic and recent compressive-sensing methods, with a 2 dB gain over OPINENet+ at a 1% CS ratio on Set11.The margin becomes more evident at low CS ratios, although DGUNet can outperform DGUNet+ when the training set is small.
Ablation Study: Performance increases with stage count, and seven stages are selected as a tradeoff between performance and computational complexity.The ablation evaluates 9, 7, 5, and 3 stages on Rain100H.
Ablation Study: Inter-stage feature fusion improves performance, more fusion scales help, spatial-adaptive fusion outperforms direct addition, and removing FGDM causes a 0.55 dB degradation on Rain100H.These ablations evaluate the proposed information pathways and flexible gradient descent module in DGUNet+.

5. Conclusion and Discussion

The paper combines model-based optimization principles with end-to-end deep learning through DGUNet. It unfolds PGD with gradient estimation and inter-stage feature pathways, and reports strong performance, interpretability, and generalizability across image-restoration tasks.

5. Conclusion and Discussion: DGUNet unfolds PGD into a deep network, integrates gradient estimation into gradient descent, and adds multi-scale spatial-adaptive inter-stage feature pathways.The design targets complex and real-world image restoration while compensating for information loss in DUNs.
5. Conclusion and Discussion: Experiments across twelve synthetic and real-world test sets demonstrate state-of-the-art performance, interpretability, and generalizability.The stated future work is supporting DGUNet on the MindSpore platform.