PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning Techniques for Inverse Problems in Imaging

Gregory Ongie, Ajil Jalal, Christopher A. Metzler, Richard G. Baraniuk, Alexandros G. Dimakis, Rebecca Willett

arXiv:2005.06001v1eess.IVcs.LGstat.ML

TL;DR

The paper addresses how deep learning can solve diverse imaging inverse problems despite ill-posed measurements and varying forward-model knowledge. It develops a taxonomy organized by forward-model use and supervised versus unsupervised training, then discusses reconstruction trade-offs, failure modes, and open problems. The resulting synthesis highlights broad potential alongside important limitations in recognizing failures and understanding untrained generative methods.

  • Problem

    Deep learning’s applicability, limitations, and foundational understanding for imaging inverse problems remain insufficiently understood.

  • Method

    The paper synthesizes deep-learning reconstruction approaches into a taxonomy based on forward-model knowledge and supervised or unsupervised training.

  • Results

    The paper identifies trade-offs including generality versus sample complexity and reconstruction speed versus accuracy across learned inverse-problem approaches.

  • Takeaways & Limitations

    The taxonomy supports comparison of reconstruction methods while highlighting the need for careful evaluation, failure-mode awareness, and further theoretical work.

  • Takeaways & Limitations

    Generative-model reconstructions can appear perceptually high quality even when they fail under heavily underdetermined measurements or outliers.

Abstract

from arXiv · show

Recent work in machine learning shows that deep neural networks can be used to solve a wide variety of inverse problems arising in computational imaging. We explore the central prevailing themes of this emerging area and present a taxonomy that can be used to categorize different problems and reconstruction methods. Our taxonomy is organized along two central axes: (1) whether or not a forward model is known and to what extent it is used in training and testing, and (2) whether or not the learning is supervised or unsupervised, i.e., whether or not the training relies on access to matched ground truth image and measurement pairs. We also discuss the trade-offs associated with these different reconstruction approaches, caveats and common failure modes, plus open problems and avenues for future work.

1 Introduction

Inverse problems reconstruct unknown signals or images from noisy observations produced by typically non-invertible forward processes. The paper introduces deep learning methods and a taxonomy for comparing reconstruction approaches, their trade-offs, and limitations.

  • Problem: Inverse problems recover an unknown image or volume from observations generated by a generally non-invertible forward process.Ill-posedness makes unique reconstruction difficult without prior knowledge about the data.
  • Applications: The framework covers tasks including deblurring, deconvolution, inpainting, compressed sensing, superresolution, MRI, CT, and radar imaging.
  • Measurement model: Measurements can be modeled using a possibly nonlinear forward operator A and additive or non-additive noise.The goal is to recover x⋆ from measurements y.
  • Conventional methods: Classical reconstruction methods use priors such as smoothness, sparsity, or geometric structure to select plausible solutions.
  • Deep learning: Deep neural networks learn regularized reconstructions across many computational imaging tasks, while deep generative models can constrain reconstructions to a learned manifold.The paper organizes these methods into a taxonomy and discusses their trade-offs and future directions.

2 Opportunities and Recent Progress

Deep learning has shown substantial potential across medical, photographic, microscopic, geophysical, and other computational imaging problems. The paper presents examples while emphasizing that the field’s applicability, limitations, and foundational understanding remain incomplete.

  • Progress and open questions: Deep learning has demonstrated enormous potential for solving diverse imaging inverse problems, but its applicability and limitations remain insufficiently understood.The authors identify opportunities for careful scientific evaluation and foundational research.
  • Medical imaging: Medical imaging methods can improve image quality and achieve orders-of-magnitude faster reconstructions than classical iterative methods.The discussion includes MRI, CT, PET, and SPECT, and cites FDA-approved deep learning reconstruction for CT.
  • Medical imaging: An undersampled MRI example compares least-squares, total generalized variation, and deep learning reconstructions using SSIM quality measures.
  • Computational photography: Deep learning supports low-light imaging and monocular depth estimation in computational photography.Networks denoise and white-balance underexposed images and produce depth maps from photographs.
  • Computational microscopy: Computational microscopy uses deep learning both to reconstruct images and to design illumination patterns and optical elements.The cited example concerns learned illumination patterns for Fourier ptychographic microscopy.
  • Other applications: Deep learning has also been proposed for geophysical imaging and other challenging problems including lensless imaging, holography, ghost imaging, scattering-media imaging, and non-line-of-sight imaging.

3 Anatomy of an Inverse Problem

An inverse problem maps an unknown image to noisy measurements through a forward operator, then seeks a plausible reconstruction using likelihoods and priors. The paper situates imaging tasks and introduces supervised and unsupervised learning categories for addressing these challenges.

  • Forward model: The measurement model represents observations as y = A(x) + ε or y = N(A(x)), with A describing the forward process and noise affecting measurements.
  • Likelihood estimation: Maximum-likelihood estimation uses p(y|x) to recover x when the noise distribution is known.
  • Limitations: Maximum-likelihood solutions may be non-unique for rank-deficient operators or highly noise-sensitive when the forward operator is poorly conditioned.
  • Prior-based reconstruction: MAP estimation incorporates prior knowledge through a prior distribution and includes Tikhonov, sparsity, and total-variation regularization.
  • Limitations: MAP methods face difficulties when noise or signal statistics are unknown, likelihoods lack closed forms, or the forward operator is unknown or only partially known.
  • Learning categories: The taxonomy distinguishes supervised methods using matched image-measurement pairs from unsupervised methods using unpaired images and measurements, images only, or measurements only.
  • Deep generative models: Deep generative models represent complex high-dimensional distributions through neural transformations from simple seed distributions.

4 Taxonomy for Learning to Solve Inverse Problems

The paper’s taxonomy compares learned inverse-problem methods by the forward-model information available and by whether training uses matched ground-truth image and measurement pairs. It is intended to clarify method families and their trade-offs.

  • Method families: Learning-based inverse methods either learn a regularizer or directly learn a mapping from measurements y to reconstructed images x̂.
  • Taxonomy axes: The taxonomy organizes methods by what is known about the forward model A and by the nature of the training data.
  • Forward-model axis: Forward-model knowledge may be available during training, introduced only at test time, or only partially known through uncertain calibration parameters.
  • Trade-offs: Using the forward model during training and testing can substantially reduce learning sample complexity.Known forward models can nevertheless be computationally prohibitive, especially when repeatedly applied during backpropagation.
  • Taxonomy overview: Taxonomy categories include major combinations of forward-model knowledge and supervised or unsupervised training, with representative examples summarized in Table 2.

4.1 Forward Model Fully Known During Training and Testing

When the forward model is fully known, supervised reconstruction networks can embed that model directly or mimic iterative optimization, while self-supervision can train from measurements when noise statistics are available.

  • Model-aware supervised reconstruction: Known forward models can be incorporated into supervised reconstruction networks that map measurements y to images x.The model or related operators, including adjoints and derivatives, can be embedded in the network architecture.
  • Approximate-inverse networks: An approximate inverse first maps measurements into the image domain, after which a trainable network removes artifacts and predicts residual content.Common approximate inverses include A⊤, A†, bicubic interpolation, and filtered back projection.
  • Unrolled networks: Unrolled methods embed A and A⊤ across multiple layers by replacing proximal operators in iterative algorithms with trainable CNNs.Step sizes and other free parameters can also be learned during training.
  • Unrolled networks: Unrolling extends beyond proximal gradient descent to ADMM, primal-dual methods, half-quadratic splitting, block coordinate descent, alternating minimization, and AMP.The approach was pioneered in a sparse coding context and later investigated for imaging inverse problems.
  • Self-supervision: Self-supervision uses measurements as a ground-truth proxy when both the forward model and noise statistics are known, with SURE generalized to other forward operators through GSURE.SURE-based training does not require access to x⋆ and has been applied to DnCNN and LDAMP networks.

4.2 Forward Model Known Only at Test Time

When the forward model is unavailable during training but known at test time, one trained model can work across forward operators, using learned denoisers or generative priors. These approaches trade retraining flexibility against optimization and computational limitations.

  • Setting and trade-offs: A single trained deep model can be reused for any forward model when A is known only at test time.This avoids training separate models for different forward operators, which is useful when ground-truth data is abundant.
  • Training strategies: Ground-truth-only training commonly learns either a proximal operator or denoiser, or a generative prior.Both approaches avoid requiring matched measurement and ground-truth pairs during training.
  • Learned denoisers: Plug-and-play and RED methods use learned or existing denoisers in place of proximal operators within iterative optimization algorithms.RED provides a framework that can use deep neural networks as denoisers.
  • Learned denoisers: LDAMP learns a denoiser for a variant of AMP and reports state-of-the-art reconstructions with a 100-fold speed improvement over other state-of-the-art methods.Its state evolution heuristic predicts reconstruction mean square error at each iteration.
  • Generative priors: CSGM searches a learned generative model’s latent space for an image that best explains the measurements, but optimizing this non-convex objective is NP-hard.The practical algorithm starts from random latent initialization and uses gradient descent or ADAM.
  • Generative priors: CSGM empirically achieves similar reconstruction quality with 5−10-fold fewer measurements than sparsity-based LASSO methods.Theoretical guarantees relate measurement counts to latent dimension, network depth, Lipschitzness, and representation error.
  • Limitations and theory: Generative-model guarantees ensure closeness to the best representation the model can achieve, while polynomial-time guarantees remain limited for general generators.Gradient-descent analysis is available under random weights and logarithmic layer expansion, but broader cases remain open.
  • Limitations and extensions: Trained generative priors require massive datasets and long training times, motivating work on untrained generative networks.The surveyed methods have also been extended to phase retrieval, blind deconvolution, seismic imaging, bilinear estimation, and 1-bit compressed sensing.

4.3 Forward Model Partially Known

With only partial knowledge of the forward model, inverse reconstruction relies on model statistics, unpaired or contaminated data, or adversarial and generative approaches. The methods differ in assumptions, flexibility, and computational cost.

  • Problem setting: Partial forward-model knowledge arises when the operator is parametric and only its parameter distribution or sufficient statistics are known.This setting requires methods that account for uncertainty in the measurement process.
  • Problem setting: Approximate knowledge of A can complicate reconstruction, while real-world paired observations reflect the true forward process during training.This allows supervised networks to exploit partial model knowledge while learning from actual observations.
  • Unpaired data: Unpaired ground-truth and measurement samples provide only marginal distributions, as in clean and motion-blurred MRI scans without correspondence.CycleGAN learns image-to-measurement and measurement-to-image mappings using cycle consistency and adversarial losses.
  • Generative priors: Generative priors can address blind deconvolution by modeling both the unknown image and blurring kernel with separate generative models.The estimates are obtained by optimizing a measurement-consistency objective and decoding the learned latent variables.
  • End-to-end reconstruction: End-to-end methods such as DeblurGAN produce crisper images from blurry inputs but require retraining after even minor changes in the forward-model distribution.Training uses synthetically blurred clean images.
  • Measurements-only learning: Learning from measurements alone is difficult, so methods commonly assume a distribution for the forward operator and knowledge of its statistics.The surveyed alternatives include supervised approaches and adversarial training.
  • Measurements-only learning: Noise2Noise learns from noisy input-output pairs without clean ground truth under the condition E[˜x|y]=x⋆.Its theoretical justification uses contaminated samples, and explicit knowledge of A’s parameters or distribution is unnecessary.
  • Adversarial training: AmbientGAN learns the measurement distribution by simulating measurements from generated images and sampled forward operators, then reconstructs through constrained least squares.After training, inference does not require distributional assumptions on A if the ground-truth distribution was learned exactly.

4.4 Unknown Forward Model

When the forward model is unknown, reconstruction generally requires matched image–measurement pairs, while black-box neural networks can learn the measurement-to-image map directly. AUTOMAP exemplifies this strategy, although its experiments used the forward model to generate training data.

  • Unknown Forward Model: Unknown or unusable forward models leave matched image–measurement pairs as the main route for supervised learning.Without paired samples, additional information about the forward model is needed to make the problem identifiable.
  • Unknown Forward Model: A black-box reconstruction network approximates the map from measurements y to images x using conventional neural-network architectures.This approach uses input and output dimensions appropriate to the measurement and image spaces.
  • Unknown Forward Model: AUTOMAP applies this black-box strategy by learning a reconstruction map through neural-network parameterizations of coordinate mappings and convolutional layers.The framework uses fully connected layers for one mapping and CNN layers for others.
  • Unknown Forward Model: AUTOMAP experiments used the known forward model to generate training data, although the framework is intended in principle to work without access to it.This distinction separates the experimental setup from the proposed capability.

5 Key Tradeoffs

Deep reconstruction methods trade generality, data efficiency, and computational cost. Decoupling learning from the forward model supports reuse across inverse problems, whereas incorporating the model can reduce sample complexity and unrolled methods can reduce reconstruction computation.

  • Sample Complexity vs. Generality: Forward-model-independent unsupervised methods can be reused across different inverse problems without retraining, giving them high generality.Generative models and denoising autoencoders are trained only on collections of images.
  • Sample Complexity vs. Generality: This generality requires learning a full image prior p(x), whereas forward-model-aware methods can learn the conditional density p(x|y).The conditional formulation uses the measurement relationship y = Ax + ϵ during learning.
  • Sample Complexity vs. Generality: In inpainting, knowing which pixels are missing narrows learning from all possible images to the distribution of missing pixels conditioned on observed pixels.Figure 10 illustrates how task-specific information can reduce the distribution that must be learned.
  • Sample Complexity vs. Generality: Conditional density estimation can use fewer samples when its smoothness α′ exceeds α and its dependence dimension d′ is much smaller than d.The cited scaling is N^−α′/(2α′+d′), compared with full-density estimation scaling N^−α/(2α+d).
  • Reconstruction Speed vs. Accuracy: Deep-learned approaches can achieve comparable accuracy with far less computation by fixing a small number of unrolled optimization blocks and learning a suitable regularizer.Preconditioners can further reduce the number of blocks and increase speed.

6 Caveats/Beware/Failure Modes

Deep reconstruction methods remain vulnerable to model mismatch, unfamiliar image features, artifacts, and opaque failure modes. These caveats motivate careful evaluation, interpretability work, robustness to outliers, and uncertainty quantification.

  • Robustness and Generalization: A forward model used at test time may differ from the one used during training, reducing robustness across scanners or other acquisition settings.The paper identifies this as a model-mismatch issue affecting learned reconstructions.
  • Robustness and Generalization: Training with discretized forward models and images can create test-time artifacts, including Gibbs ringing in MRI.The paper calls this practice the “inverse crime.”
  • Robustness and Generalization: Methods trained on representative MRI data can recover an unseen square feature but also produce non-trivial artifacts beside it.Figure 12 shows artifacts to the feature’s left and right.
  • Interpretability: Deep image priors and supervised models remain difficult to interpret because their success and inference behavior lack a solid theoretical framework.A smooth-signal bias is offered as a current hypothesis for deep image prior behavior, not as a complete explanation.
  • Artifacts: Realistic-looking reconstructions can contain nonexistent features, creating problematic artifacts for downstream tasks such as tumor classification.The paper emphasizes that perceptual realism does not guarantee image fidelity.
  • Failure Recognition: Generative-model reconstruction can return a perceptually high-quality image even when heavily underdetermined or outlier-contaminated data cause failure.The paper contrasts this hard-to-recognize failure with the visibly non-sparse output of Lasso and motivates robustness or uncertainty quantification.

7 Open Problems and Future Directions

Future work includes designing trainable measurement and optical systems, extending methods beyond imaging, and clarifying how reconstruction approaches behave across data, forward-model, and uncertainty settings.

  • Control over forward model design: Deep learning can optimize measurement patterns and differentiable optical systems for task-specific imaging design.Examples include MRI k-space sampling, single-pixel-camera patterns, optical filters, diffractive elements, and depth-related imaging systems.
  • Extensions to other application domains: The tutorial’s themes may extend to other inverse-problem domains, while architecture choices and algorithmic trade-offs remain open questions.Additional physical constraints or side information may require frameworks beyond knowledge encoded solely in the forward model A.
  • Unlearned Methods: Untrained generative models achieve strong empirical reconstruction results, but their operating conditions, regularization, and complexity remain poorly understood.Deep Decoder avoids early stopping through underparameterization, while DIP uses early stopping to prevent fitting noise.
  • Transfer learning: Transfer learning addresses inverse problems with limited real training data or simulated data that differ from the target application domain.The challenge is especially relevant in medical imaging and astronomy.
  • Nonlinear inverse problems: Nonlinear inverse problems can show empirical success, but general solution strategies and trade-offs for nonlinear forward models remain insufficiently understood.Phase retrieval is highlighted as an example where several works report strong empirical results.
  • Uncertainty quantification: Uncertainty quantification is essential for applications such as medical diagnosis, yet most reviewed learning approaches do not provide uncertainty estimates.Bayesian approaches address this gap by estimating posterior distributions or posterior-derived statistics.
Loading 2005.06001v1…