PaperlayerView exact revision on arXiv

Source-linked AI summary

A Review of Convolutional Neural Networks for Inverse Problems in Imaging

Michael T. McCann, Kyong Hwan Jin, Michael Unser

arXiv:1710.04011v1eess.IVcs.CV

TL;DR

Inverse imaging requires reconstructing images from sparse or noisy measurements, while CNN-based methods raise questions about training data, architecture, and optimization. This review synthesizes recent CNN work and theoretical perspectives, finding improvements over direct and some regularized methods, with MRI results matching state of the art and offering running-time advantages. It also identifies nonconvex learning and reproducibility challenges.

  • Problem

    Sparse or noisy measurements make direct inversion artifact-prone or unstable, while inverse-problem CNN training lacks the straightforward oracle labels available in tasks such as classification.

  • Method

    The paper reviews CNN applications and theory for imaging inverse problems, organizing evidence around training-set construction, network architecture, and learning formulation and optimization.

  • Results

    CNN methods improve over direct and some regularized reconstructions in medical imaging, while MRI reconstruction reaches state-of-the-art performance with running-time advantages.

  • Takeaways & Limitations

    CNN architectures can be reused across applications, expanded as memory grows, and run fast after training, but reported inverse-problem gains are more limited than their classification impact.

Abstract

from arXiv · show

In this survey paper, we review recent uses of convolution neural networks (CNNs) to solve inverse problems in imaging. It has recently become feasible to train deep CNNs on large databases of images, and they have shown outstanding performance on object classification and segmentation tasks. Motivated by these successes, researchers have begun to apply CNNs to the resolution of inverse problems such as denoising, deconvolution, super-resolution, and medical image reconstruction, and they have started to report improvements over state-of-the-art methods, including sparsity-based techniques such as compressed sensing. Here, we review the recent experimental work in these areas, with a focus on the critical design decisions: Where does the training data come from? What is the architecture of the CNN? and How is the learning problem formulated and solved? We also bring together a few key theoretical papers that offer perspective on why CNNs are appropriate for inverse problems and point to some next steps in the field.

I. INTRODUCTION

The review surveys CNN-based image reconstruction for inverse problems, defining a constrained scope intended to support meaningful generalizations. It contrasts objective-function reconstruction with learned parametric reconstruction and highlights training-data, modeling, and optimization challenges.

  • Scope: The review covers recent CNN applications that recover images from noisy measurements, excluding detection, segmentation, classification, and quality assessment.Its literature search targets applications with at least three peer-reviewed papers from the last ten years.
  • Inverse problems: Inverse imaging seeks a reconstruction operator R that recovers an image x from measurements y produced by an imaging operator H.In X-ray CT, x represents attenuation, H models the source and detector physics, and y is a measured sinogram.
  • Reconstruction formulations: The objective-function approach models the imaging system and combines a data-fidelity term with a regularizer such as total variation or sparsity promotion.For CT, filtered back projection is an example of a direct inverse, while total variation penalizes large image gradients.
  • Inverse problems: Direct inverses become artifact-prone or unstable when measurements are sparse or noisy, motivating regularized reconstruction methods.Measurements may be costly because of acquisition time or patient X-ray exposure, encouraging collection of as few measurements as possible.
  • Reconstruction formulations: The learning approach fits a parametric reconstruction from ground-truth image–measurement pairs and then applies it to new measurements.It avoids handcrafting the forward model, cost function, regularizer, and optimizer, but requires training data and typically solves a nonconvex problem.
  • Reconstruction formulations: Learning and objective-function formulations form a spectrum, with the learning formulation strictly more general and able to include objective formulations as special cases.Choosing which objective-formulation components to retain is a critical design decision.

B. Convolutional Neural Networks

CNNs represent reconstruction algorithms through alternating convolutions and pointwise nonlinearities, with filter weights learned by gradient-based optimization. Their fast execution motivated applications beyond classification, while inverse-problem gains remained an open empirical question.

  • Architecture: CNN reconstruction algorithms use sequences of filtering operations alternating with simple nonlinear operations, parameterized by learned filter weights.A typical CNN fixes the function class and minimizes over its filter weights.
  • Practical advantages: CNN execution uses small convolutions and pointwise nonlinearities, making trained reconstructions fast and amenable to GPU acceleration.The chain-rule gradient also involves small convolutions, enabling efficient gradient-descent learning.
  • Historical motivation: The review asks whether CNNs can produce similarly large gains for inverse problems, including an order-of-magnitude or 20 dB improvement in denoising.It addresses this question by surveying reported CNN-based image-reconstruction results.

III. CURRENT STATE OF PERFORMANCE

Across denoising, deblurring, super-resolution, and medical imaging, CNN methods generally report modest improvements, although medical-imaging comparisons are difficult because experimental setups vary. CNNs may nonetheless offer practical advantages in reuse, scalability, and running time.

  • Denoising: Denoising results generally improve by less than 1 dB over contemporary non-CNN methods, including BM3D.Reported average PSNR values include 28.5 on the Berkeley Segmentation Dataset and 30.4 dB on a standard test-image experiment.
  • Deblurring and super-resolution: Deblurring gains were around 1 dB over comparison methods, with a further improvement of around 1 dB reported in later work.
  • Deblurring and super-resolution: Super-resolution improvements ranged from less than 0.5 dB to around 1.5 dB in PSNR over earlier comparison baselines.Video super-resolution reports included successive improvements of about 0.5 dB PSNR.
  • Medical imaging: Medical-imaging comparisons are difficult because studies use widely varying experimental setups, but reported CNN gains range from about 0.5 dB to 1–4 dB.MRI work reached state-of-the-art performance with running-time advantages.
  • Practical considerations: Once trained, CNN-based methods can run in less than one second per image and can be reused or expanded as memory grows.

IV. DESIGNING CNNS FOR INVERSE PROBLEMS

The survey organizes CNN design for inverse problems around the learning equation, separating training-set creation, architecture design, and learning formulation and solution.

  • CNN design decisions are organized into training-set creation, network-architecture design, and learning-problem formulation and solution.

A. Training Set

Training data is central but difficult to obtain for inverse imaging because exact oracle outputs are often unavailable, especially for medical problems. Researchers therefore use synthetic corruption or fully sampled measurements when feasible.

  • Inverse-problem training lacks the human-style oracle labels available in classification, making exact 3D ground truths for physical CT phantoms impractical.
  • Training-set choices constrain CNN architecture because network inputs and outputs must match the dimensions of the measurements and target images.
  • Denoising, super-resolution, and deblurring can generate training pairs from known forward transformations such as noise corruption or downsampling.Residual learning can use the noise itself as the oracle output.
  • Medical-imaging methods commonly train on sparse-measurement inputs paired with reconstructions from fully sampled measurements.Examples include under-sampled k-space paired with fully sampled MRI reconstructions and low-view CT paired with high-view CT.

2) Preprocessing:

Preprocessing often composes a direct inverse with a CNN, allowing the network to refine an initial reconstruction rather than learn the full inverse physics from raw measurements. Practical choices also include dynamic-range scaling and training-set management.

  • Preprocessing: In a 50-view CT example, the CNN reconstruction preserves more ground-truth texture and yields a significant SNR increase relative to FBP and regularized reconstruction.The ground truth uses 1000 views; comparison reconstructions use 50 views.
  • Preprocessing: Many applications preprocess measurements with a direct inverse before CNN processing, forming Rθ◦˜H−1.Examples include interpolation in super-resolution, FBP in CT, and the inverse Fourier transform in MRI.
  • Preprocessing: Without preprocessing, the CNN must learn the inverse problem’s underlying physics, whereas preprocessing leverages established direct-inverse engineering.
  • Practical preparation: Training-image intensity or contrast should be scaled when dynamic ranges vary substantially, preventing a small subset from dominating the training error.
  • Practical preparation: Patch size, network depth, filter size, training-set size, and regularization jointly affect practical CNN design and generalization.Larger filters and deeper networks require larger training patches, while small datasets limit performance despite regularization or validation.

B. Network Architecture

CNN architectures for inverse problems range from simple convolutional stacks to designs adapted from existing networks or derived by unrolling iterative algorithms. Learning objectives commonly use likelihood-based formulations, with Euclidean loss implicitly assuming Gaussian noise.

  • Architecture choices: CNN architecture choices include layer count, filter sizes, nonlinearities, and parameter count, ranging from 8,032 to roughly one hundred thousand parameters.The survey denotes the architecture by R_θ and the learned weights by θ.
  • Architecture choices: Simple convolutional stacks provide a baseline whose capacity can be adjusted through layers, channels, and filter sizes.Using small filters can support deeper networks.
  • Adapted and modular designs: Existing CNN architectures can be adapted to inverse problems, such as changing a biomedical segmentation network’s outputs from two image classes to one reconstructed image.Modular networks can also be connected end-to-end for tasks such as four-times super-resolution.
  • Unrolling and hybrid designs: Unrolling converts iterations of an optimization algorithm into network layers, replacing fixed filters with learned ones and often producing CNNs through alternating linear and nonlinear steps.Unrolled MRI reconstruction achieved state-of-the-art results and improved running time; the original algorithm is a special case when training succeeds.
  • Learning formulation: Learning can be formulated by maximizing conditional likelihood, while maximum a posteriori learning incorporates prior information about CNN parameters.Under Gaussian noise, Euclidean loss without regularization is equivalent to maximum likelihood; Gaussian weight and noise models yield Euclidean weight regularization.

D. Optimization

CNN training minimizes a chosen objective using practical optimization methods, while initialization and regularization address nonconvexity and overfitting. Validation-based stopping and dropout are among the described safeguards.

  • Training objective: After fixing the learning objective, CNN training minimizes it using software libraries that efficiently train user-defined architectures.The survey treats this minimization as a practical overview despite its theoretical depth.
  • Gradient-based optimization: Stochastic gradient descent estimates gradients from random training subsets, reducing computation while retaining an approximate gradient.Momentum can further tune the process by combining gradients from previous iterations.
  • Initialization: Because learning is nonconvex, initialization can affect the converged minimum, while good initializations can improve convergence speed.Pretrained networks and weights initialized from corresponding known filters are popular choices.
  • Overfitting control: Overfitting is a serious risk for networks with potentially millions of parameters, motivating training-set augmentation and validation procedures.The supplied passage introduces splitting data into optimization and validation sets.
  • Overfitting control: Validation monitoring can stop training when performance drops, and dropout randomly deletes network units during training as a regularization method.Dropout is motivated as a weighted average over possible parameter settings.

V. THEORY

Theoretical perspectives explain CNN suitability for inverse imaging through local approximation and connections to iterative reconstruction algorithms. These connections also identify conditions under which CNN reconstruction may be unsuitable.

  • Motivation: The review assembles theoretical perspectives because CNN performance is strong while the reasons for that performance remain mostly unanswered.The cited perspectives begin to explain why CNNs fit inverse problems in imaging.
  • Approximation: CNNs can be viewed patch by patch, with each input patch mapped to its output patch by a fully connected network.This supplies a local approximation perspective related to universal approximation results.
  • Algorithmic connection: Unrolling links CNNs to established iterative algorithms whose alternating linear and pointwise nonlinear steps resemble CNN operations.Spline-theoretic results can also approximate proximal operators used in traditional iterative methods.
  • Scope conditions: Conditions on the forward model ensure that an iterative method’s linear step is convolutional, and the surveyed inverse problems satisfy these conditions.The theory predicts that some problems, such as structured illumination microscopy, may not be amenable to CNN reconstruction.

3) Invariance:

CNNs offer theoretical potential for additional invariances beyond scattering transforms, but their use in inverse problems remains constrained by reproducibility and learning-robustness concerns.

  • Invariance: Scattering transforms provide translation and small-deformation invariance, while CNNs generalize them with potential additional invariances.
  • Algorithm Descriptions and Reproducibility: Incomplete reporting of network architectures, training, and evaluation procedures prevents reliable quantitative comparisons and impedes reproducibility.
  • Robustness of Learning: CNN learning is nonconvex, so initialization and optimization choices may affect results and complicate comparisons between methods.
  • Robustness of Learning: An l1-trained network can outperform an l2-trained network even on the l2 cost, which the cited authors attribute to l2 learning becoming stuck in a local optimum.
  • Robustness of Learning: Larger networks trained with more data may mitigate learning instability, although further work on learning stability is needed.
  • Robustness of Learning: Sensitivity to training sets, architectures, and optimization can enable test-set overfitting through iterative design choices.

3) Can We Trust the Results?:

Trust in CNN-generated reconstructions depends on testing sensitivity, cross-dataset generalization, and relevance to the ultimate application rather than relying only on image-quality metrics.

  • CNN reconstructions remain nonlinear and highly complex, so their sensitivity to input noise is an important trust consideration.Data augmentation during training can help achieve robustness to small input changes.
  • Evaluation should target the reconstruction’s ultimate application, such as diagnosis or biological quantification, rather than only SNR or SSIM.This is especially relevant because CNNs are often treated as black boxes and produce plausible-looking images that may conceal uncertainty.
  • CNN research in imaging has focused mainly on image-valued measurements with simple models, while CT and MRI have received less attention because their models are more complicated.The survey expects further work in CT, MRI, and other modalities.
  • Future systems should combine CNNs with underlying physics and direct or iterative inversion techniques.Possible designs include embedding inverse operators in the network, using back-projected measurements, or approximating slow high-quality reconstruction methods.

2) Cross-Task Learning:

Cross-task learning could let CNNs trained for one imaging setting transfer to related tasks, reducing retraining when parameters change, while multidimensional imaging remains constrained by computation and emerging alternatives seek more perceptual outputs.

  • 2) Cross-Task Learning:: Cross-task learning trains an algorithm on one dataset and deploys it on a different but related task.This can avoid costly retraining when noise levels or image dimensions change.
  • 2) Cross-Task Learning:: Most CNN approaches to modern 3D or 3D+time reconstruction still use 2D inputs and outputs.The text attributes this to the field’s 2D emphasis and practical constraints.
  • 2) Cross-Task Learning:: Current GPUs’ maximum 24 GB physical memory limits training large networks with 3D inputs and outputs.Model parallelism and data parallelism are proposed to address this constraint.
  • 2) Cross-Task Learning:: Generative adversarial networks may offer a way to overcome current limits in supervised learning by training generator and discriminator networks in competition.The generator learns a mapping between training samples while the discriminator distinguishes generated outputs from real data.
  • 2) Cross-Task Learning:: Perceptual loss produces better-looking results than standard Euclidean loss but does not typically improve SNR.Its acceptance in medical imaging remains uncertain because those applications require quantitative accuracy.
Loading 1710.04011v1…