PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Convolutional Neural Network for Inverse Problems in Imaging

Kyong Hwan Jin, Michael T. McCann, Emmanuel Froustey, Michael Unser

arXiv:1611.03679v1cs.CV

TL;DR

The paper addresses imaging quality and acquisition-related artifacts and risks by proposing FBPConvNet, which combines direct inversion with a deep convolutional network. It reports compelling results on synthetic and real data, favorable comparison with iterative reconstruction on realistic datasets, and sub-second per-image computation after training.

  • Problem

    Imaging quality must be maintained while avoiding long acquisitions that may cause motion artifacts, patient discomfort, or patient harm.

  • Method

    FBPConvNet combines direct inversion with a deep convolutional network for inverse problems.

  • Results

    The method produced compelling results on synthetic and real data, compared favorably with state-of-the-art iterative reconstruction on two realistic datasets, and reconstructed one image in under a second after training.

  • Takeaways & Limitations

    FBPConvNet is feasible for sparse-view CT and offers favorable reconstruction performance with low post-training computation time.

  • Takeaways & Limitations

    A major limitation is the lack of transfer between datasets.

Abstract

from arXiv · show

In this paper, we propose a novel deep convolutional neural network (CNN)-based algorithm for solving ill-posed inverse problems. Regularized iterative algorithms have emerged as the standard approach to ill-posed inverse problems in the past few decades. These methods produce excellent results, but can be challenging to deploy in practice due to factors including the high computational cost of the forward and adjoint operators and the difficulty of hyper parameter selection. The starting point of our work is the observation that unrolled iterative methods have the form of a CNN (filtering followed by point-wise non-linearity) when the normal operator (H*H, the adjoint of H times H) of the forward model is a convolution. Based on this observation, we propose using direct inversion followed by a CNN to solve normal-convolutional inverse problems. The direct inversion encapsulates the physical model of the system, but leads to artifacts when the problem is ill-posed; the CNN combines multiresolution decomposition and residual learning in order to learn to remove these artifacts while preserving image structure. We demonstrate the performance of the proposed network in sparse-view reconstruction (down to 50 views) on parallel beam X-ray computed tomography in synthetic phantoms as well as in real experimental sinograms. The proposed network outperforms total variation-regularized iterative reconstruction for the more realistic phantoms and requires less than a second to reconstruct a 512 x 512 image on GPU.

I. INTRODUCTION

The paper examines unresolved links between iterative reconstruction and CNNs, then proposes combining direct inversion with a CNN for normal-convolutional inverse problems. It demonstrates the approach on low-view CT using synthetic and real data.

  • Iterative reconstruction with regularization has become a dominant approach for radiation-based imaging inverse problems.
  • Deep learning has shown strong results in image analysis and regression-type inverse problems, motivating CNN-based reconstruction methods.
  • Practical and theoretical questions remain about when CNNs outperform iterative reconstruction and which learned components produce the gains.
  • The proposed method targets inverse problems whose normal operator H*H is a convolution, a class including MRI, X-ray CT, and diffraction tomography.
  • The method combines fast approximate direct inversion with a CNN using residual and multilevel learning for low-view CT reconstruction.
  • The network produced images representing complex textures well and was compared with state-of-the-art regularized iterative reconstruction on synthetic and real CT data.

II. INVERSE PROBLEMS WITH SHIFT-INVARIANT NORMAL OPERATORS

The paper identifies inverse problems with shift-invariant normal operators as a broad class in which iterative algorithms use convolutional filtering and point-wise nonlinearities. This structure motivates CNNs as an alternative solver.

  • For these problems, iterative solution methods require repeated convolutions and point-wise nonlinearities.
  • The considered class includes denoising, deconvolution, and MRI, CT, and diffraction tomography reconstruction.
  • The convolutional structure is known for MRI and CT and has previously been exploited to design fast algorithms.
  • The section introduces the class to motivate a method combining direct and iterative approaches.

A. Theory

The theory characterizes operators whose normal operator is a convolution and illustrates the result with the continuous X-ray transform. In that example, the normal operator has a frequency-domain convolution filter.

  • A convolution is represented in the Fourier domain as F* M_hat h F, with M_hat h acting as a multiplication operator.
  • Theorem 1 gives conditions involving an isometry, multiplication, and reversible change of variables that ensure H*H is a convolution.
  • The discrete construction H_d = S H_c Q also yields a convolutional normal operator when H_c Qf is bandlimited.
  • For the continuous 2D X-ray transform, coordinate changes and Fourier operations map spatial functions to polar-frequency representations and sinograms.
  • The X-ray transform satisfies R*R as a convolution with frequency response h_hat(ω) = 1/∥ω∥.

B. Direct Inversion

For normal-convolutional operators, direct inversion can be performed through filtering and back projection, but ill-posedness amplifies noise. Discrete sampling and interpolation also affect reconstruction accuracy and runtime.

  • The reconstruction problem recovers an image f from measurements g = Hf using direct inverses based on the normal operator.
  • One direct method applies the inverse filter of H*H after back projection, equivalent to deconvolution in reconstruction space.
  • A second method inverts H in the measurement domain before back projection, using a multiplication operator.
  • When T is a Fourier transform, the second inverse is filtering followed by back projection; otherwise it remains filtering-like in the associated transform domain.
  • In practice, discrete measurements make alternative direct-inversion forms approximate, affecting reconstruction accuracy and runtime.
  • The inverse filter greatly amplifies noise, so practical reconstruction requires some smoothing.

C. Iterative Inversion

Ill-posed imaging inverse problems require regularization because direct inversion is highly sensitive to measurement noise. The paper frames iterative sparse-reconstruction methods as filtering followed by point-wise nonlinearities when the normal operator is convolutional.

  • Direct inversion is unstable for ill-posed imaging problems because measurement noise can severely perturb the solution.
  • Regularization such as total variation or l1 sparsity is used to stabilize inverse reconstruction.
  • The discrete inverse problem represents measurements as y = Hx and seeks a reconstruction through sparse transform coefficients a with x = Wa.
  • Because the regularized formulation lacks a closed-form solution, it is typically solved iteratively.
  • For normal-convolutional forward models and convolutional W, ISTA iterates filter the estimate, add a data-dependent bias, and apply soft-thresholding.
  • Other methods, including ADMM, FISTA, and SALSA, use the same basic combination of filtering and point-wise nonlinearities.

III. PROPOSED METHOD: FBPCONVNET

The proposed FBPConvNet uses a direct inversion followed by a CNN for normal-convolutional inverse problems. For CT, filtered back projection incorporates the physics and supplies a useful input for artifact-removal learning.

  • CNNs are motivated by the filtering-plus-nonlinearity structure of iterative methods for normal-convolutional inverse problems.
  • FBPConvNet applies discretized filtered back projection to measurements, then feeds the result to a CNN trained against suitable ground-truth images.
  • The approach is intended in principle for all normal-convolutional inverse problems, but this work focuses on CT reconstruction.
  • Performing FBP before the CNN incorporates physical knowledge and provides a warm start, simplifying learning compared with direct measurement-to-image regression.
  • For CT, using FBP avoids requiring the CNN to learn the transformation between polar sinogram coordinates and Cartesian image coordinates.
  • Efficient direct inversions are available for normal-convolutional inverse problems, although FBP itself is specific to CT.

B. Deep Convolutional Neural Network Design

The network builds on U-net with multilevel decomposition, multichannel filtering, and residual learning rather than explicitly imitating an iterative algorithm. It is evaluated for sparse-view X-ray CT against FBP and TV-based iterative reconstruction.

  • B. Deep Convolutional Neural Network Design: The CNN explores a state-of-the-art architecture rather than directly imitating an unfolded iterative method.
  • B. Deep Convolutional Neural Network Design: The proposed architecture is based on U-net, whose multilevel decomposition provides larger effective filters in deeper layers.
  • B. Deep Convolutional Neural Network Design: Larger effective filters help represent the potentially non-compact support of H*H and its inverse, such as in CT.
  • B. Deep Convolutional Neural Network Design: Multichannel filtering increases expressive power and generalizes the channel-like structure associated with transform subbands or split variables.
  • B. Deep Convolutional Neural Network Design: A skip connection makes the network learn the input-output difference, mitigates vanishing gradients, and improves performance over the corresponding network without the skip connection.
  • B. Deep Convolutional Neural Network Design: The implementation uses zero-padding and a final convolution that reduces 64 channels to one output image.
  • B. Deep Convolutional Neural Network Design: Experiments compare FBPConvNet with FBP alone and a TV-regularized ADMM reconstruction using FFT-based filtering.
  • B. Deep Convolutional Neural Network Design: Sparse-view evaluation subsamples full-view sinograms, using full-view FBP as ground truth and applying both methods to the same subsampled data.

A. Data Preparation

The evaluation uses two synthetic CT datasets and one real experimental dataset. Sinograms are downsampled to sparse-view settings for testing reconstruction methods.

  • The study evaluates the method on three datasets: two synthetic datasets and one derived from real experiments.
  • The ellipsoid dataset contains 500 synthetic images with randomly varied ellipse intensity, size, and location.
  • Ellipsoid sinograms have 729 detector pixels and 1,000 views and are generated using the analytical X-ray transform of an ellipse.
  • The biomedical dataset contains 500 real in-vivo CT images from the Mayo Clinic Low-dose Grand Challenge database.
  • Biomedical sinograms also have 729 pixels and 1,000 views and are generated with the Matlab radon function.
  • The experimental dataset contains 377 sinograms acquired at the TOMCAT beam line of the Swiss Light Source.
  • Each experimental sinogram is 1493 pixels by 721 views and represents one z-slice from a rat brain.
  • Synthetic sinograms are subsampled by factors of 7 and 20 to produce 143- and 50-view reconstructions, while real data use 145 and 52 views.

B. Training Procedure

FBPConvNet is trained on paired low-view and full-view FBP images, then evaluated against TV reconstruction using SNR and sparse-view ellipsoidal reconstructions. On ellipsoidal data, both methods reduce artifacts at 143 views, while at 50 views FBPConvNet retains some artifacts and TV achieves higher average testing SNR.

  • Training Procedure: The CNN is trained with low-view FBP images as inputs and full-view FBP images as targets, enabling use on real CT data without oracle reconstructions.The implementation uses MatConvNet, with training and evaluation performed on a Titan Black GPU.
  • Training Procedure: The TV comparison uses parameters optimized on the training set, reflecting hand tuning in practice; learning these parameters from data remains an open question.The authors note that selecting TV parameters from training data could be a fairer comparison, but instead use golden-section search.
  • Training Procedure: SNR compares reconstructed images with oracle images, and higher SNR indicates better reconstruction.The metric is defined using the oracle x and reconstructed image x̂.
  • A. Ellipsoidal Dataset: At sevenfold downsampling, sparse-view FBP produces line artifacts, while TV and FBPConvNet significantly reduce them with visually indistinguishable results.Figure 3 evaluates ellipsoidal reconstructions from 143 views.
  • A. Ellipsoidal Dataset: At twentyfold downsampling, both methods reduce stronger FBP artifacts, but FBPConvNet retains some artifacts and has lower average testing SNR than TV.Figure 4 evaluates the more severely undersampled ellipsoidal case from 50 views.

B. Biomedical Dataset

The FBPConvNet preserved fine structures and complex textures more effectively than TV reconstruction on realistic biomedical and experimental datasets. It also reconstructed 512 × 512 images in under a second, but requires dataset-specific retraining and assumes real-valued data handling.

  • Results: The FBPConvNet preserved fine details in the biomedical and experimental datasets, improving SNR relative to iterative methods.The improvement was attributed to preserving fine image details, whereas iterative methods explicitly impose regularization.
  • Computational cost: 400–500 ms was the reported GPU computation time for FBPConvNet reconstruction of a 512 × 512 image.About 200 ms was used for FBP and 200∼300 ms for the CNN, compared with around 7 minutes for iterative reconstruction after parameter selection.
  • Limitations: The method has limited transfer between datasets: changing dimensions or subsampling factors requires retraining the network.A network trained on seven-times subsampled data retained many artifacts when applied to twenty-times subsampled data.
  • Limitations: Applying the method to MRI or DT is experimentally non-trivial because complex-valued measurements require a CNN architecture that handles complex values.The paper also identifies the need for large training datasets and a high-quality iterative comparison algorithm.

VII. CONCLUSION

The paper proposes FBPConvNet, a deep convolutional network combining filtered back projection with a multiresolution residual CNN for biomedical inverse problems. It reports favorable comparisons with state-of-the-art iterative reconstruction and sub-second per-image computation after training.

  • The paper proposes FBPConvNet, a deep convolutional network for biomedical inverse problems that combines FBP with a multiresolution CNN.The CNN uses a U-net structure with residual learning.
  • The CNN architecture is motivated by the convolutional structure of biomedical inverse problems, including CT, MRI, and DT.
  • The paper identifies conditions on a linear operator ensuring that its normal operator is a convolution, supporting CNN suitability for this inverse-problem subclass.
  • The proposed method demonstrated compelling results on synthetic and real data and compared favorably with state-of-the-art iterative reconstruction on two more realistic datasets.
  • Under a second per image, the trained proposed network reconstructs one image after training.
Loading 1611.03679v1…