Source-linked AI summary
Adversarial Regularizers in Inverse Problems
Sebastian Lunz, Ozan Öktem, Carola-Bibiane Schönlieb
TL;DR
Inverse problems commonly rely on model-based variational regularization, while data-driven methods face challenges with complex forward operators and limited training data. The paper learns a neural-network regularization functional by distinguishing ground-truth images from unregularized reconstructions and then uses it in the variational problem. Experiments demonstrate potential on denoising and computed tomography, including competitive LIDC reconstructions trained with unsupervised data.
Problem
Deep-learning approaches can become infeasible for complex forward operators and limited training data, while high-dimensional learned regularization functionals exceed the computational feasibility of small admissible function sets.
Method
The framework uses a neural network as a variational regularization functional, training it as a critic to distinguish ground-truth images from unregularized reconstructions.
Results
The framework demonstrates potential for denoising and computed tomography, with unsupervised adversarial regularization reaching 32.5 PSNR and .946 SSIM in the reported comparison.
Takeaways & Limitations
The proposed algorithm can be trained with only unsupervised data and produced reconstructions on LIDC with the same or superior visual quality as supervised approaches.
Takeaways & Limitations
Directly learning the measurement-to-image mapping can be infeasible for complex forward operators and very limited training data, settings typical of medical imaging.
Abstract
from arXiv · showhide
Inverse Problems in medical imaging and computer vision are traditionally solved using purely model-based methods. Among those variational regularization models are one of the most popular approaches. We propose a new framework for applying data-driven approaches to inverse problems, using a neural network as a regularization functional. The network learns to discriminate between the distribution of ground truth images and the distribution of unregularized reconstructions. Once trained, the network is applied to the inverse problem by solving the corresponding variational problem. Unlike other data-based approaches for inverse problems, the algorithm can be applied even if only unsupervised training data is available. Experiments demonstrate the potential of the framework for denoising on the BSDS dataset and for computed tomography reconstruction on the LIDC dataset.
1 Introduction
The paper proposes replacing hand-crafted variational regularizers with neural networks trained to distinguish ground-truth images from unregularized reconstructions. The framework combines variational inverse-problem formulations with data-driven learning and targets high-dimensional regularization functionals.
- Variational regularization combines forward-operator knowledge with an image-prior functional to reconstruct images from inverse-problem measurements.
- Deep learning has been explored for inverse problems, but efficiently using the forward operator and noise model without relearning physical structure remains difficult.
- The proposed framework replaces the hand-crafted regularization functional with a neural network within the variational formulation.
- The training algorithm uses Wasserstein generative adversarial model ideas, training the network as a critic that distinguishes ground-truth images from unregularized reconstructions.
- The paper reports theoretical properties and demonstrates the algorithm on denoising and computed tomography.
2 Background
Inverse problems recover images from noisy measurements, with applications including computer vision and medical imaging. Variational regularization provides a model-based solution, while deep-learning approaches face challenges with complex operators and limited training data.
- Problem formulation: Inverse problems recover an image x from a measurement y through a forward operator and noise term.The operator maps the image space X to the measurement space Y.
- Problem formulation: Denoising, inpainting, MRI, and CT can all be formulated as inverse problems with task-specific forward operators.Denoising uses the identity, inpainting uses a projection, MRI uses the Fourier transform, and CT uses the ray transform.
- Data-driven approaches: Directly learning the mapping y →x can work for denoising and inpainting but becomes infeasible with complex forward operators or very limited training data.Limited training data is typical in medical-imaging applications.
- Data-driven approaches: Alternative data-driven methods include pseudo-inverse preprocessing, iterative neural networks, and learned proximal operators.These approaches are designed to address inverse problems involving complex forward operators.
- Variational regularization: Variational regularization recovers x from a single measurement y by enforcing measurement consistency and incorporating prior knowledge through a functional f.The regularization functional is usually hand-crafted, with TV and TGV producing piecewise constant and piecewise linear images, respectively.
3 Learning a regularization functional
The paper learns a neural-network regularization functional by discriminating ground-truth images from unregularized reconstructions, enabling scalable variational regularization with unsupervised data. Its analysis shows that gradient descent moves the reconstruction distribution closer to the ground-truth distribution and, under a data-manifold assumption, yields a distance-based regularizer.
- Existing bilevel methods for learning regularization functionals are computationally feasible only for small admissible classes and do not scale to high-dimensional parameterizations.
- The proposed training algorithm learns a regularization functional that scales to high-dimensional parameter spaces by discriminating noisy from ground-truth images.The functional can be represented by a neural network ΨΘ, with network parameters defining the admissible class.
- Training uses independent samples from the ground-truth-image and measurement marginals, placing the method in an unsupervised-learning setting.No paired samples from the joint image-measurement distribution are required.
- 3.1 Regularization functionals as critics: The critic loss compares typical ground-truth and noisy-reconstruction samples while adding a gradient penalty to enforce one-Lipschitz continuity.The loss is based on Wasserstein adversarial training, and the penalty is evaluated along lines connecting samples from the two distributions.
- 3.2 Distributional Analysis: A gradient-descent step over the learned functional produces a reconstruction distribution that is closer in Wasserstein distance to the ground-truth distribution than the noisy distribution.This creates an incentive for minimizers of the variational problem to align with the ground-truth image distribution.
- 3.3 Analysis under data manifold assumption: Under the weak data-manifold assumption, the learned regularization functional takes the form of the l2 distance to the data manifold.The distance-function form provides meaningful gradients, unlike a characteristic-function regularizer.
4 Stability
The paper derives a stability estimate for the adversarial regularizer algorithm, replacing the usual lower-boundedness assumption with a 1-Lipschitz condition on the regularizer.
- The adversarial regularizer algorithm has a stability estimate following the theory of classical variational problems.
- The analysis does not assume that the regularization functional f is bounded from below.Instead, it assumes that f is 1-Lipschitz.
- For convergent data measurements, minimizers have a weakly convergent subsequence whose limit minimizes ∥Ax −y∥2 + λf(x).
5 Computational Results
The method estimates the regularization parameter from the noise level without further tuning and is evaluated for BSDS denoising and LIDC CT reconstruction.
- 5.1 Parameter estimation: The regularization parameter λ is estimated from the noise level and used in all experiments without further tuning.The estimate relies on the ground-truth critical-point assumption and unit-norm gradients of the regularization functional.
- 5.2 Denoising: On BSDS denoising, the adversarial regularizer outperforms classical variational methods in all reported quality measures.Its visual quality is comparable to supervised data-driven algorithms despite using no supervised training data.
- 5.3 Computed Tomography: On LIDC CT reconstruction, total variation is outperformed by a large margin across all noise-level categories.The comparison includes total variation, post-processing, RED, and the proposed regularizers.
6 Conclusion
The paper combines neural-network regularization with variational inverse-problem methods and demonstrates competitive reconstructions using unsupervised data on LIDC CT.
- The proposed algorithm uses a neural network as the regularization functional for solving inverse problems.
- The algorithm can be trained when only unsupervised training data is available.This targets settings where appropriate supervised training data is unavailable.
- The variational framework incorporates the forward operator and noise model, supports stability theory, and adapts to noise levels by changing λ without retraining.
- On the LIDC dataset, the method produces reconstructions of the same or superior visual quality to supervised approaches, while classical methods are outperformed by a large margin.
- The approach is particularly suited to medical imaging, where few training samples and scarce ground-truth measurements can make supervised algorithms impossible to train.
7 Extensions
The paper describes local regularizers and recursive training as extensions intended to reduce data requirements or improve performance, while recursive training requires delicate sample selection.
- Local Regularizers: Local regularizers act on small pixel patches and average their values across patches.Convolutional layers followed by global average pooling can implement this design.
- Local Regularizers: Local regularizers can generate many training samples from one image, enabling training with even less data.
- Recursive Training: Recursive training adds partially reconstructed images to the samples used to train the regularization functional.This lets the network learn from its own outputs.
- Recursive Training: Initial implementations report an additional performance boost from recursive training, but selecting which images to add is delicate.
A Stability Theory
Under the stated assumptions, the variational reconstruction problem has minimizers and is weakly stable under norm-convergent perturbations of the data. The distance-to-manifold regularizer also satisfies the needed weak lower-semicontinuity property.
- Assumptions: The analysis assumes weak lower semicontinuity and 1-Lipschitz continuity of f, weak-to-weak continuity of A, and one of two additional conditions.These assumptions are described as standard in classical inverse-problem stability theory, except for replacing a lower-bound condition on f with 1-Lipschitz continuity.
- Distance-to-Manifold Regularization: For the distance function d_M to the data manifold M, weak lower semicontinuity holds, satisfying the relevant regularity requirement.The paper establishes this property specifically when the regularization functional is based on distance to the manifold.
- Existence of Minimizers: Under Assumption 3, a minimizer exists for the variational objective.The proof uses bounded minimizing sequences, extraction of a weakly convergent subsequence, and weak lower semicontinuity.
- Weak Stability in Data Term: If y_n converges to y in norm, a sequence of minimizers has a weakly convergent subsequence whose limit minimizes the objective for y.The result establishes weak stability with respect to perturbations in the data term.
B Implementation details
The experiments use a compact convolutional network and gradient-descent optimization for adversarial regularization, with UNet-style supervised comparators and PDHG for total variation. Visual comparisons cover BSDS denoising and LIDC CT reconstruction across noise settings.
- Adversarial Regularizer: The adversarial regularizer uses an 8-layer convolutional network with four stride-2 convolutions, leaky ReLU activations, two dense layers, RMSProp, and fixed-step gradient descent.The regularization parameter follows the heuristic specified in the paper.
- Neural Baselines: The Denoising Neural Network and Post-Processing baselines use UNet-style architectures with four down-sampling and four transposed-convolution up-sampling layers connected by skip connections.These comparison networks use leaky ReLU activations, Adam optimization, and an ℓ2 ground-truth training loss.
- Total Variation: Total variation is optimized with primal-dual hybrid gradient descent, while its regularization parameter is selected by line search for the best PSNR.This provides the implementation protocol for the model-based comparison.
- BSDS Denoising: BSDS denoising comparisons display ground truth, noisy input, TV, Denoising Neural Network, and Adversarial Reg. from left to right.The figure is intended for visual comparison across the five reconstruction or reference columns.
- LIDC CT Reconstruction: LIDC CT comparisons display ground truth, FBP, TV, Post-Processing, and Adversarial Reg. from left to right for high- and low-noise settings.The low-noise figure additionally places the reconstruction sinogram below the image comparisons.
- Synthetic Ellipse Data: On synthetic ellipse CT data, Adversarial Regularizers produce fewer artifacts around small-angle intersections of different domains than TV.This figure isolates an artifact pattern relevant to the comparison between the learned regularizer and total variation.