Source-linked AI summary
Deep Unfolding of a Proximal Interior Point Method for Image Restoration
Carla Bertocchi, Emilie Chouzenoux, Marie-Caroline Corbineau, Jean-Christophe Pesquet, Marco Prato
TL;DR
Variational restoration can incorporate prior knowledge, but solving and tuning its models can be costly, while deep networks offer less control over outputs. The paper introduces iRestNet, an unfolded proximal interior point network with learned layer parameters; on image-deblurring benchmarks it compares favorably with other methods.
Problem
Variational restoration incorporates priors but can require costly solving and parameter tuning, whose best choices may depend on the data.
Source: §1 · p. 2Method
iRestNet unfolds a proximal interior point algorithm, using a logarithmic barrier to incorporate constraints and learning the stepsize, barrier parameter, and regularization weight across layers.
Source: §1 · p. 3Results
On BSD500, iRestNet’s mean SSIM exceeds the other methods except for MotionA, where it ranks second after IRCNN; it also compares favorably on Flickr30.
Source: §6.6 · p. 21Takeaways & Limitations
After training, iRestNet can process test images with short execution time and without user parameter search, unlike traditional variational methods.
Source: §1 · p. 3Handcrafted IPM parameter sequences may reduce efficiency and versatility for perceptual image quality, while statistical selection of λ can substantially increase computational cost.
Source: §2.4 · pp. 6–7
Abstract
from arXiv · showhide
Variational methods are widely applied to ill-posed inverse problems for they have the ability to embed prior knowledge about the solution. However, the level of performance of these methods significantly depends on a set of parameters, which can be estimated through computationally expensive and time-consuming methods. In contrast, deep learning offers very generic and efficient architectures, at the expense of explainability, since it is often used as a black-box, without any fine control over its output. Deep unfolding provides a convenient approach to combine variational-based and deep learning approaches. Starting from a variational formulation for image restoration, we develop iRestNet, a neural network architecture obtained by unfolding a proximal interior point algorithm. Hard constraints, encoding desirable properties for the restored image, are incorporated into the network thanks to a logarithmic barrier, while the barrier parameter, the stepsize, and the penalization weight are learned by the network. We derive explicit expressions for the gradient of the proximity operator for various choices of constraints, which allows training iRestNet with gradient descent and backpropagation. In addition, we provide theoretical results regarding the stability of the network for a common inverse problem example. Numerical experiments on image deblurring problems show that the proposed approach compares favorably with both state-of-the-art variational and machine learning methods in terms of image quality.
1. Introduction
The introduction motivates combining variational priors with deep learning and proposes iRestNet, which unfolds a proximal interior point algorithm while retaining explicit optimization components and learning parameters across layers.
1. Introduction
- Variational restoration incorporates prior information through constraints or regularizers, but solving the resulting problem and tuning its parameters can be time-consuming and data-dependent.Early stopping may help, but choosing its stopping time requires an oracle such as SURE.Source: §1 · p. 2
- Deep networks perform well on inverse problems, but their black-box character limits control over outputs, while unfolding combines learned parameters with an iterative method.Unlike image-specific early stopping, unfolded networks use a layer count selected during offline training and fixed for test images.Source: §1 · pp. 2–3
- iRestNet unfolds a proximal interior point algorithm, using a logarithmic barrier for feasible iterates and learning the stepsize, barrier parameter, and regularization weight per layer.The barrier supports incorporating prior knowledge while allowing gradient-based training.Source: §1 · pp. 3–4
- Unlike approaches that learn some functions or operators, iRestNet keeps the proximity operator and regularization term explicit, linking the network to its source algorithm.Its training objective may target perceptual quality, so the trained output need not minimize the objective that inspired the architecture.Source: §1 · pp. 3–4
2. Proximal interior point algorithm
The section formulates image restoration as a constrained penalized optimization problem and develops a forward–backward proximal interior-point scheme. It motivates learning the algorithm’s parameters because heuristic settings and statistical regularization-parameter selection can be inefficient or costly.
2.1. Variational formulation and notation
- The restoration objective combines data fidelity and λ-weighted regularization over inequality-defined C, with twice-differentiable terms assumed for backpropagation.C is assumed to have nonempty interior, and existence of solutions is guaranteed if the penalized objective is coercive or C is bounded.Source: §2.1 · pp. 4–5
2.2. Interior point approaches
- Interior-point methods replace constrained optimization with barrier-based subproblems as µ vanishes; the proximal IPM avoids the matrix inversions required by standard IPMs.The proximal forward–backward approach is described as a competitive solver for inverse problems.Source: §2.2 · p. 5, §2.3 · p. 6
2.3. Proposed iterative schemes
- Algorithm 2 adds a forward step because Algorithm 1’s summed proximity operator often lacks a closed form, leaving only the barrier proximity operator to compute.The authors note that no convergence study for Algorithm 2 was available in the interior-point literature.Source: §2.3 · p. 6
2.4. Limitations
- Heuristic µ and γ settings may reduce efficiency and versatility, while statistical selection of λ substantially increases computational cost.The proposed response is to learn the stepsize, barrier, and regularization parameters in a supervised fashion.Source: §2.4 · pp. 6–7
3. Proximity operator of the barrier
The section derives logarithmic-barrier proximity operators and their derivatives for three common constraint types. These expressions provide the quantities needed to train the proposed network through gradient backpropagation.
3. Proximity operator of the barrier
- For three common constraint types, the section gives barrier proximity operators and their input and parameter derivatives for backpropagation.The derivatives are with respect to the input x, barrier parameter µ, and stepsize γ.Source: §3 · p. 7
3.1. Affine constraints
- For a half-space constraint, the barrier proximity operator has an explicit expression, with its Jacobian and µ- and γ-gradients also derived.The proposition states the operator and derivatives for γ > 0 and µ > 0.Source: §3.1 · pp. 7–8
3.2. Hyperslab constraints
- For a hyperslab aᵀx ∈ (b_m, b_M), the operator uses the unique root κ in that interval of a cubic equation, and its derivatives are provided.The three cubic roots can be computed using the Cardano formula.Source: §3.2 · pp. 8–9
3.3. Bounded ℓ2-norm
- For a bounded ℓ2-norm constraint, the operator is expressed using the unique κ in [0, √α) solving a cubic equation, with its Jacobian and parameter gradients derived.The derivation handles a ball centered at c, reducing the nonzero-center case to the c = 0 case.Source: §3.3 · pp. 9–12
4. iRestNet architecture
iRestNet unfolds a proximal interior-point update into a finite network whose layerwise structures estimate algorithm parameters. Its differentiable barrier proximity operator enables gradient-based training, while task-dependent regularization and post-processing structures accommodate restoration settings.
4.1. Overview
- iRestNet unfolds Algorithm 2 for K iterations, with λ untied across layers and each layer estimating µ_k, γ_k, and λ_k before applying the update.A post-processing layer Lpp may follow the K updates.Source: §4.1 · p. 12
4.2. Hidden structures
- The stepsize is constrained to be positive using Softplus, while the barrier parameter is estimated by two convolutional and average-pooling layers followed by a fully connected layer.The authors note Softplus’s gradient is never strictly zero, which helps propagate gradients through the network.Source: §4.2 · p. 13
- The regularization-parameter structure depends on R; experiments estimate noise using median-filtered wavelet diagonal coefficients, allowing noise levels to vary within a reasonable range.For deblurring, the task-dependent post-processing layer uses convolutions, residual learning, batch normalization, and dilation to remove remaining artifacts.Source: §4.2 · p. 13
4.3. Differential calculus
- Backpropagation uses the chain rule through differentiable steps, with explicit proximity-operator derivatives supplied for common barrier constraints.The data-fidelity and regularization functions are assumed twice differentiable.Source: §4.3 · p. 13
5. Network stability
For quadratic data-fidelity and regularization, the section relates iRestNet to feedforward networks and gives sufficient conditions for averagedness, which bounds sensitivity to input perturbations.
5. Network stability
- The stability analysis focuses on a subclass with quadratic data-fidelity and regularization functions.It aims to give explicit conditions ensuring robustness.Source: §5 · p. 14
5.1. Relation to generic deep neural networks
- iRestNet has a feedforward-network form, with proximal operators serving as activation functions.Its layer operators can also be written as a proximal activation operator plus a bias.Source: §5.1 · p. 15
5.2. Preliminary results
- Proposition 5 gives sufficient conditions in three cases, based on the eigenvalues of the symmetric weight matrix and network parameters.The cases constrain the sum of the largest and smallest eigenvalues alongside θ, α, and the number of layers.Source: §5.2 · p. 15
5.3. Averaged operator
- An α-averaged operator is a blend of the identity and a nonexpansive operator, with smaller α corresponding to greater stability.The averaging constant also controls an upper bound on the effect of an input perturbation.Source: §5.3 · p. 16
5.4. Robustness of iRestNet to an input perturbation
- Under shared diagonalization of H⊤H and D⊤D, Theorem 1 gives sufficient conditions for iRestNet to be α-averaged, enabling an output-perturbation bound.The theorem's conditions can be checked using its eigenvalue expressions.Source: §5.4 · pp. 16–17
6. Numerical experiments
The experiments evaluate iRestNet on non-blind color image deblurring, with results generally favoring it over variational and machine-learning competitors on BSD500 and Flickr30.
6.1. Problem formulation
- The task is non-blind RGB deblurring with a known blur kernel and additive Gaussian noise, formulated using smoothed total variation and pixel-intensity bounds.The experiments set pixel bounds to 0 and 1 and use a smoothing parameter of 0.01, which was not fine-tuned.Source: §6.1 · p. 17
6.2. Network characteristics
- The network uses 40 unfolded layers; its noise-level estimate does not require prior noise-level knowledge, and its post-processing module uses nine dilated convolutional layers with residual learning.The dilation varies across layers to widen the receptive field, and the post-processing module includes batch normalization.Source: §6.2 · p. 18
6.3. Dataset and experimental settings
- Training uses 1,200 RGB images and evaluates on 200 BSD500 and 30 Flickr30 images across multiple blur and noise configurations.The training set combines 200 BSD500 training images and 1,000 COCO images; test images are center-cropped to 256 × 256.Source: §6.3 · pp. 18–19
6.4. Training
- Training proceeds greedily through the first 30 layers, then jointly trains the remaining layers and post-processing module; Adam minimizes negative SSIM.Training takes approximately 3 to 4 days per blur kernel, while test time is about 1.4 seconds per image on a GeForce GTX 1080 GPU.Source: §6.4 · pp. 19–20
6.5. Evaluation metrics and competitors
- Performance is measured by SSIM against a variational comparator with per-image SSIM-optimal parameters and deep or unfolded restoration methods, with some methods unavailable for selected settings.The variational comparator is an unrealistic upper-bound scenario because it assumes a perfect estimator of the error.Source: §6.5 · pp. 20–21
6.6. Results and discussion
- On BSD500, iRestNet has the highest average SSIM for every degradation model except MotionA, where IRCNN ranks first; it performs well on most individual images.IRCNN is the most competitive method on MotionA, while iRestNet requires no user tuning of regularization parameters during training.Source: §6.6 · p. 21, Figure 5 · p. 22, Table 2 · p. 23
- On Flickr30, which supplied no training images, iRestNet compares favorably with the other approaches on this test set with different image statistics.This evaluates transfer of the trained networks to a test set with statistics different from the training set.Source: §6.6 · p. 21, Table 3 · p. 23
- Visual examples show better recovery of snake and caterpillar skin details and leaves, while GaussianB results avoid background artifacts seen with PDHG and IRCNN.The authors suggest the latter competitors are not robust to a small change in noise level.Source: §6.6 · pp. 21, 23
7. Conclusion
iRestNet unfolds a proximal interior point algorithm, using a logarithmic barrier to enforce constraints and computable proximity-operator derivatives to support training. The conclusion reports favorable image-restoration comparisons and identifies blur-kernel-specific training as a limitation.
7. Conclusion
- A logarithmic barrier incorporates hard constraints into the network, while proximity-operator derivatives for three standard constraint types are well-defined and computable.The approach can also extend to multiple regularization functions or multi-variable parameterized penalizations.Source: §7 · p. 23
- Visual results and SSIM are shown for different methods on BSD500 images degraded by GaussianB and Square, and a Flickr30 image degraded by MotionB.Source: Figure 6 · p. 24, Figure 7 · p. 24, Figure 8 · p. 25
- iRestNet performs favorably against state-of-the-art variational and machine-learning methods on image restoration, without requiring noise-level knowledge or hand-selected regularization parameters.The network must be trained for a given blur kernel.Source: §7 · p. 24
- Figure 9 plots the estimated stepsize, barrier parameter, and regularization weight across network layers for the Figure 6 image.Source: Figure 9 · p. 25
- The authors propose extending the method to incompletely known observation models, including blind or semi-blind deconvolution.This follows the stated limitation that iRestNet must be trained for a given blur kernel.Source: §7 · pp. 24–25