PaperlayerView exact revision on arXiv

Source-linked AI summary

Image Reconstruction: From Sparsity to Data-adaptive Methods and Machine Learning

Saiprasad Ravishankar, Jong Chul Ye, Jeffrey A. Fessler

arXiv:1904.02816v3eess.IVcs.LGstat.ML

TL;DR

Medical image reconstruction must recover high-quality images from limited or corrupted measurements while balancing model assumptions, noise, artifacts, and computational demands. This paper reviews the progression from analytical and model-based reconstruction to sparsity, low-rank, and data-driven learning methods. It synthesizes their connections, applications, trade-offs, and open questions.

  • Problem

    High-quality reconstruction from limited or corrupted measurements remains important for reducing scan time, radiation dose, and effects of patient motion.

  • Method

    The paper reviews sparsity, low-rank, and machine-learning reconstruction models, including their mathematical underpinnings, connections, algorithms, applications, and limitations.

  • Results

    The review identifies reconstruction methods spanning simple model-based regularization, improved sparsity and low-rank models, and sophisticated data-driven or adaptive approaches.

  • Takeaways & Limitations

    Sparsity and low-rank models support reconstruction from limited or corrupted data, while learning-based methods provide data-driven or adaptive alternatives with distinct trade-offs.

Abstract

from arXiv · show

The field of medical image reconstruction has seen roughly four types of methods. The first type tended to be analytical methods, such as filtered back-projection (FBP) for X-ray computed tomography (CT) and the inverse Fourier transform for magnetic resonance imaging (MRI), based on simple mathematical models for the imaging systems. These methods are typically fast, but have suboptimal properties such as poor resolution-noise trade-off for CT. A second type is iterative reconstruction methods based on more complete models for the imaging system physics and, where appropriate, models for the sensor statistics. These iterative methods improved image quality by reducing noise and artifacts. The FDA-approved methods among these have been based on relatively simple regularization models. A third type of methods has been designed to accommodate modified data acquisition methods, such as reduced sampling in MRI and CT to reduce scan time or radiation dose. These methods typically involve mathematical image models involving assumptions such as sparsity or low-rank. A fourth type of methods replaces mathematically designed models of signals and systems with data-driven or adaptive models inspired by the field of machine learning. This paper focuses on the two most recent trends in medical image reconstruction: methods based on sparsity or low-rank models, and data-driven methods based on machine learning techniques.

I. INTRODUCTION

Medical image reconstruction progressed from fast analytical methods to iterative, compressed-sensing, and learning-based approaches that model imaging physics, data properties, or image structure. This review focuses on sparsity, low-rankness, and machine-learning methods for reconstructing limited or corrupted measurements.

  • Analytical methods use simple imaging-system models, including filtered back-projection for CT and inverse Fourier methods for MRI and CT.
  • Iterative reconstruction incorporates imaging physics, measurement statistics, and image priors to estimate images while reducing noise and artifacts.
  • Compressed-sensing methods accommodate reduced MRI or CT sampling to reduce scan time or radiation dose, assuming sparsity in a transform domain or dictionary.
  • Learning-based methods replace mathematically designed signal and system models with data-driven or adaptive models, learned from datasets or jointly with reconstruction.
  • Clinical iterative methods for PET, SPECT, and CT use relatively simple edge-preserving regularizers that moderately encourage gradient sparsity.

III. SPARSITY USING MATHEMATICAL MODELS

Mathematical sparsity models reconstruct images by encouraging transform coefficients, gradients, or dictionary representations to be sparse. The section also highlights prior-image constraints and the limitation of relying on operators designed mathematically rather than adapted from data.

  • Analysis regularization assumes that an image becomes sparse after applying a spatial operator T, with finite differences and wavelets among the most common choices.
  • The ℓ1 sparsity penalty is a convex relaxation of the ℓ0 support count, while nonconvex penalties with 0 < p < 1 can more closely approximate ℓ0.
  • Prior-image constrained compressed sensing modifies reconstruction to encourage similarity to an available prior image.
  • Synthesis models represent an image as x = Dz, a sparse combination of dictionary atoms, and coincide with analysis formulations only under specific operator and dictionary conditions.
  • Traditional sparsity models design operators such as T and D mathematically rather than computing them directly from training data or patient-specific measurements.

IV. LOW-RANK MODELS

Low-rank models exploit redundancy in image data, especially correlated dynamic sequences, while structured low-rank methods connect image-domain sparsity to Fourier-domain Hankel rank. These models support reconstruction from undersampled or corrupted measurements, with trade-offs such as temporal pooling artifacts.

  • Low-rank assumptions are particularly useful for dynamic or time-series data because image sequences tend to be correlated over time.
  • Dynamic MRI methods reshape space-time data into Casorati matrices or use local patches, while later approaches combine low-rank and sparse components.
  • Pooling dynamic MRI data over time can cause poor temporal resolution and artifacts, whereas model-based techniques can improve temporal or spatial resolution in undersampled settings.
  • L+S reconstruction decomposes the object as x = xL + xS and uses low-rank and sparsity penalties, with accurate recovery requiring distinguishable components and appropriately incoherent acquisition.
  • Structured low-rank Hankel methods exploit the duality between spatial-domain sparsity and spectral-domain Hankel-matrix rank through annihilating-filter relationships.
  • Measurement-domain optimization estimates fully sampled Fourier data before a simple inverse Fourier transform, enabling flexible single-coil and multichannel MRI reconstruction.

V. DATA-DRIVEN AND LEARNING-BASED MODELS

Data-driven and learning-based reconstruction methods replace fixed mathematical models with adaptive models, including partially data-adaptive patch sparsity schemes. The reviewed approaches span learned dictionaries, transforms, and other machine-learning models.

  • Learning-based reconstruction uses data-driven or adaptive image models and includes methods for solving often nonconvex learning-based problems efficiently.
  • Early compressed-sensing MRI methods used fixed wavelet, total-variation, or contourlet transforms, whereas later methods incorporated directional patches or block matching.
  • Patch-based directional wavelets adaptively rearrange and transform image patches, using an analysis-sparsity regularizer formed from their ℓ1 norms.
  • PANO preestimates similar patch groups through block matching and penalizes group sparsity in a known transform domain.

B. Synthesis Dictionary Learning-Based Approaches for Reconstruction

Synthesis dictionary learning models image patches as sparse combinations of learned atoms and can jointly estimate the image and dictionary from limited measurements. MRI methods evolved from expensive, nonconvergent sparse coding toward efficient exact-update schemes with convergence guarantees and improved image quality.

  • Synthesis dictionary model: Synthesis dictionary learning approximates each image patch as a sparse combination of dictionary atoms, forming a union-of-subspaces model.Different patches may use different subsets of dictionary columns.
  • Dictionary learning for MRI: DL-MRI jointly estimates the image and patch dictionary from limited measurements without requiring training data.The learned dictionary adapts to the underlying image content, but the optimization is highly nonconvex.
  • Dictionary learning for MRI: K-SVD-based DL-MRI significantly improved image quality over compressed-sensing MRI methods using nonadaptive wavelets and total variation.The method was slow because of repeated sparse-coding steps and lacked convergence guarantees.
  • Extensions: Dictionary learning has also been applied to dynamic MRI, parallel MRI, PET, and CT reconstruction.In sparse-view CT, jointly learning the dictionary was more promising than pre-learning and fixing it when sufficient views were available.
  • Recent efficient methods: SOUP-DIL uses aggregate ℓ0 sparsity penalties to allow variable sparsity levels across patches and optimizes dictionary columns and code rows by exact closed-form updates.Its block-coordinate descent scheme decomposes DZ into outer products and applies thresholding to coefficient rows.
  • Recent efficient methods: SOUP-DIL was proved to converge to critical points and improved image quality over several reconstruction schemes.The scheme addresses the earlier DL-MRI method’s expensive inexact updates and absent convergence analysis.

5) Alternative Convolutional Dictionary Model:

Convolutional dictionary models represent images directly through convolutions of filters and sparse coefficient maps, offering an alternative to patch-based synthesis models. Their main limitation is inadequate representation of very low-frequency content, while transform learning provides an efficient alternative with closed-form updates.

  • Convolutional dictionary model: Convolutional synthesis dictionaries represent an image as a sum of convolutions between dictionary filters and sparse coefficient maps.This model is distinct from patch-based synthesis dictionaries.
  • Convolutional dictionary model: Convolutional dictionary models cannot represent very low-frequency image content well and therefore require preprocessing to remove it.Their use for biomedical inverse problems remains an open research direction.
  • Transform learning: Sparsifying transform learning models image patches through approximately sparse transformed coefficients with an explicit transform-domain error.The transform model generalizes analysis dictionaries by allowing Wf = z + e.
  • Transform learning: Transform-learning reconstruction alternates updates of sparse codes, the transform, and the image, with closed-form thresholding for sparse coding.The square-transform regularizer prevents trivial solutions and helps control the transform’s condition number.
  • Transform learning: A continuation strategy can vary sparsity across iterations, enabling faster initial artifact removal followed by reduced bias.The resulting scheme was shown to be much faster than the earlier DL-MRI method.
  • Applications: Square transform learning has been applied to CT and low-dose CT, including a shifted-Poisson likelihood and a pre-learned fixed transform.Other work has considered overcomplete or tall transforms for potential reconstruction use.

3) Learning Rich Unions of Transforms for Reconstruction:

Rich unions of learned transforms model diverse image textures, features, and edges by assigning patches to specialized transforms. These models support reconstruction across CT and MRI, while STROLLR additionally combines non-local low-rank structure with local transform sparsity.

  • Learning Rich Unions of Transforms for Reconstruction: A union of K learned transforms clusters image patches so each class uses the transform that best matches its texture or structure.The UNITE formulation jointly learns transforms, patch clusters, sparse codes, and the reconstructed image, using an efficient BCD algorithm with convergence guarantees.
  • Learning Rich Unions of Transforms for Reconstruction: UNITE-MRI improved image quality over square transform learning for undersampled k-space MRI measurements.
  • Learning Rich Unions of Transforms for Reconstruction: A pre-learned union-of-transforms model enabled clustering-based low-dose 3D CT reconstruction, with PWLS-ULTRA removing noise and preserving edges better than FDK and PWLS-EP.The example uses K = 15 at I0 = 5 × 10^3 incident photons per ray.
  • Learning Rich Unions of Transforms for Reconstruction: Structured transform learning can impose flipping and rotation invariance, producing a flexible model organized around directional properties.Images with more directional features are better modeled by these learned transforms.
  • Learning Complementary Models – Low-rank and Transform Sparsity: STROLLR combines non-local self-similarity with local patch sparsity, jointly exploiting low-rank structure and transform learning.Block matching forms matrices of similar patches, while the learned higher-dimensional transform jointly sparsifies them.
  • Learning Complementary Models – Low-rank and Transform Sparsity: STROLLR-MRI achieved better compressed-sensing MRI image quality than several methods, including supervised deep-learning ADMM-Net.Its block matching and sparsity mechanism implicitly groups similar patches.

D. Online Learning for Reconstruction

Online learning adapts reconstruction models sequentially as dynamic measurements arrive, while filterbank and multilayer transform interpretations connect learned sparse models to efficient reconstruction architectures. These approaches support temporal adaptivity, low-memory computation, and model-based multilayer processing.

  • D. Online Learning for Reconstruction: Online reconstruction adapts models over time from incoming measurement sequences to track dynamic objects and aid reconstruction.Dynamic MRI methods process frame mini-batches with sliding windows and update the objective using recent measurements while adapting the dictionary.
  • D. Online Learning for Reconstruction: Warm starts let online algorithms run only a few optimization iterations per new window, while cumulatively updated matrices keep dictionary updates memory-efficient.
  • D. Online Learning for Reconstruction: Online learning algorithms were reported as more efficient and more effective than batch learning-based techniques for dynamic MRI.Batch methods iteratively learn and reconstruct from all k-t space measurements.
  • Connections with Convolutional Filterbanks: Transform sparse coding on regularly spaced patches is equivalent to convolution with learned filters followed by thresholding of coefficient maps.Transform learning therefore has an equivalent interpretation as learning convolutional sparsifying filters.
  • Multi-layer Transform Learning: Deep residual transforms extend transform learning with filtering, nonlinearities, pooling, stacking, and linear decoder backpropagation in an unsupervised model-based framework.Pooling residual maps along the filter-channel dimension provides robustness to noise and data corruptions.
  • Multi-layer Transform Learning: The unrolled reconstruction model uses K layers corresponding to iterations, alternating convolutional decorruption with thresholding and a system-model image update solved using FFTs or CG.The system block enforces the imaging forward model after adding a fixed bias term.

F. Physics-Driven Deep Training of Transform-Based Reconstruction Models

Physics-driven deep training unfolds iterative reconstruction algorithms into neural-network layers while learning transform-model components from data. These approaches combine model-based updates with learned filtering, thresholding, and synthesis operations.

  • Physics-driven deep training: Unrolled physics-driven networks map iterative reconstruction steps to layers containing system-model updates and learned decorruption modules.Each decorruption module applies transform filtering, thresholding, matched synthesis filtering, and summation; the system-model block can use FFTs or conjugate gradients.
  • Physics-driven deep training: Blind compressed-sensing schemes learn transform filters during iterative block coordinate descent rather than from a separate training dataset.Because filters are learned on the fly, they can change across iterations or network layers.
  • Physics-driven deep training: Matched synthesis and transform filters improved image quality in learned physics-driven networks, where thresholding remains central to exploiting sparsity.Once learned, these reconstruction models can be applied efficiently using convolutions, thresholding, and least-squares-type updates.
  • Broader development: Deep learning became prominent in reconstruction across CT, MRI, PET, ultrasound, and optics, with commercial CT systems receiving FDA approval by mid-2019.The surge was supported by public datasets, GPU infrastructure, accessible toolboxes, industrial involvement, and rapid open publication.

A. Categories of the existing approaches

Deep-learning reconstruction methods differ by where learning is performed and how explicitly they retain measurement consistency. The reviewed realizations include image-domain, hybrid-domain, unrolled optimization, CNN-prior, plug-and-play, and deep-image-prior approaches.

  • Image-domain learning: Image-domain methods first generate artifact-corrupted images analytically, then train neural networks to learn artifacts relative to clean labels.Low-dose and sparse-view CT examples use FBP reconstructions from noisy or sparse sinograms as network inputs.
  • Image-domain learning: FBPConvNet shows that image-domain CNNs can arise by unrolling sparse recovery when the forward model’s normal operator is a convolution.The cited class includes MRI, parallel-beam X-ray CT, and diffraction tomography.
  • Hybrid-domain learning: Hybrid-domain methods impose data consistency during neural-network training and inference, with physics-driven networks including full data-fidelity updates in every layer.Other hybrid approaches use CNN penalties within MBIR or replace iterative denoising steps with neural denoisers.
  • Unrolled optimization: Unrolling approaches convert iterative algorithms such as ISTA, Landweber, ADMM, primal-dual, and projected-gradient methods into trainable neural networks.Learned quantities include weight matrices, thresholds, convolutional operators, activation derivatives, and regularization parameters.
  • Deep image prior: Deep image prior uses a parameterized neural network to represent the reconstruction, with inputs ranging from noise vectors to simultaneously acquired MRI data for PET reconstruction.The network is parameterized by θ, while the input choice depends on the application.

3) AUTOMAP:

AUTOMAP learns a direct measurement-to-image mapping, while related reconstruction variants learn in sensor or transform domains and address limited availability of clean reference data. These methods span data-driven mappings, domain-specific residual learning, and reference-free alternatives.

  • AUTOMAP: AUTOMAP learns a direct mapping from measurement space to image space using a fully connected layer followed by convolutional layers.The fully connected layer creates high memory requirements, limiting current applications to small MRI reconstruction problems.
  • Sensor-domain learning: Sensor-domain learning trains neural networks to interpolate or denoise measurements before producing an image-domain output.A low-dose CT example learns an end-to-end sinogram-to-image network whose output acts as a data-driven ramp filter.
  • Transform-domain learning: WavResNet extends low-dose CT learning to wavelet-domain residual prediction, where choosing an appropriate transform domain supports efficient learning.The figure caption contrasts full-dose FBP, quarter-dose FBP, and WavResNet denoising applied to 25% dose FBP images.
  • Reference-data constraints: Supervised reconstruction commonly learns mappings from noisy or accelerated inputs to noiseless, high-dose, or fully sampled labels.Such labels can be difficult or impossible to acquire directly, especially when paired low- and high-dose exposures pose patient risks.
  • Reference-free learning: GAN-based low-dose CT denoising avoids matched high-dose data by matching probability distributions, but careless training can generate spurious artifacts.CycleGAN variants use cyclic and identity losses to address this sensitivity; other approaches can operate without reference data or with limited references.
  • Interpretability and representation learning: Deep-learning reconstruction faces concerns about neural-network black-box behavior and whether reported improvements are clinically real or cosmetic.The paper frames deep learning as representation learning that can avoid the expensive optimization required by classical sparse representations.

2) Deep Neural Networks as Combinatorial Representation Learning:

Deep neural networks can be interpreted as adaptive framelet representations whose bases depend on learned filters and input-dependent activation patterns. ReLU nonlinearities partition input space into regions sharing distinct linear representations, yielding large adaptive expressivity.

  • Adaptive framelet representation: Deep convolutional framelet theory interprets a deep neural network as a framelet representation whose frame basis is learned from training data.Follow-up work shows that this representation can adapt automatically to different input signals in real time.
  • Encoder-decoder structure: An encoder-decoder CNN maps an input signal into a feature space and uses a decoder to produce an output in the original signal dimension.The encoder and decoder are modeled symmetrically with multiple filter channels, pooling operations, and convolutional layers.
  • Frame representation: The encoder-decoder output admits a frame-basis and dual-frame representation whose matrices depend on layer operators and ReLU activation patterns.The associated diagonal matrices contain binary values determined by ReLU outputs from preceding convolution steps.
  • Frame conditions: Without ReLU nonlinearities, suitable frame conditions for pooling, unpooling, and filters can ensure perfect signal reconstruction.Because practical neural networks should transform inputs into different outputs, perfect reconstruction is not itself the practical objective.
  • Combinatorial representation: Each layer’s ReLU activation pattern partitions feature space into non-overlapping regions, with inputs in one region sharing a linear representation but differing across regions.The resulting combinatorial expansion makes the representation explicitly dependent on the input.
  • Computational implications: Input adaptivity provides a computational advantage over classical representation learning methods that rely on expensive optimization.The learned filter sets determine the representations, while exponentially large expressivity is identified as another potential advantage of deep networks.

VII. OPEN QUESTIONS AND FUTURE DIRECTIONS

The paper identifies unresolved theoretical, computational, and evaluation challenges in learning-based reconstruction, while pointing toward more adaptive and integrated imaging systems.

  • Evaluation: Task-based metrics are needed because traditional measures such as mean-squared error may not predict clinical-task performance.Objective image-quality assessment remains an ongoing research area.
  • Theory: Theoretical conditions guaranteeing accurate and stable recovery in learning-based setups remain insufficiently understood.Open questions concern learned models, cost functions, initialization, and sampling, as well as desirable CNN filters, nonlinearities, and physics integration.
  • Online learning: Online learning faces a trade-off between faster reconstruction with simpler models and image quality.Future methods should balance model richness, minibatch runtime, convergence over time, and image quality.
  • Generalization: Different learning-based approaches lack a rigorous comparison across signal-to-noise ratios, dose levels, and undersampling regimes.The paper contrasts modest-data adaptive models with supervised methods that may generalize less reliably or produce spurious reconstructions.
  • Learned sampling: Learned MRI sampling can improve image quality over conventional strategies, but current optimization may be computationally expensive with unknown convergence behavior.Efficient sampling-learning algorithms with guarantees remain a future direction.
  • Future systems: Future smart imaging systems may jointly learn sampling and reconstruction models from large datasets and real-time patient inputs.The stated goals include rapid or low-dose imaging and efficient, effective underlying models.

VIII. CONCLUSIONS

The paper surveys medical image reconstruction from analytical and model-based methods through sparsity, low-rankness, adaptive models, and machine learning, emphasizing advances, connections, limitations, and future directions.

  • Scope: The survey traces reconstruction from analytical and simple model-based methods to sparsity, low-rankness, data-driven models, and machine learning.It uses modality-specific examples and discussion to examine method behavior and limitations.
  • Synthesis: The paper discusses different regimes of adaptivity and learning and connections among learning-based models and reconstruction methods.These connections are considered alongside the field’s growing interest in modeling, learning-based imaging, and smart imaging systems.
  • Outlook: Challenges, open questions, and future directions remain for modeling and learning-based medical imaging.The conclusion places these issues within the field’s continuing growth and interest in smart imaging systems.
Loading 1904.02816v3…