PaperlayerView exact revision on arXiv

Source-linked AI summary

Physics-Driven Deep Learning for Computational Magnetic Resonance Imaging

Kerstin Hammernik, Thomas Küstner, Burhaneddin Yaman, Zhengnan Huang, Daniel Rueckert, Florian Knoll, Mehmet Akçakaya

arXiv:2203.12215v3eess.IVcs.CVcs.LGeess.SPphysics.med-ph

TL;DR

Computational MRI involves inverse problems with linear and nonlinear forward models, and the article reviews how physics-driven learning incorporates MRI physics into reconstruction. It characterizes these methods as state of the art and notes their potential relevance to translational and downstream medical-imaging applications.

  • Problem

    Computational MRI reconstruction involves inverse problems with linear and nonlinear forward models, including workflows that use raw k-space data.

    Source: §I · p. 4

  • Method

    The article reviews physics-driven learning approaches for computational MRI, including methods involving raw k-space data.

    Source: §I · p. 4, §III · p. 7

  • Results

    The article describes physics-driven deep learning techniques as the current state-of-the-art methods for computational MRI.

    Source: §VI · p. 29

  • Takeaways & Limitations

    These methods are starting to enter translational and clinical settings and may affect downstream tasks in the medical-imaging pipeline.

    Source: §VI · p. 29

Abstract

from arXiv · show

Physics-driven deep learning methods have emerged as a powerful tool for computational magnetic resonance imaging (MRI) problems, pushing reconstruction performance to new limits. This article provides an overview of the recent developments in incorporating physics information into learning-based MRI reconstruction. We consider inverse problems with both linear and non-linear forward models for computational MRI, and review the classical approaches for solving these. We then focus on physics-driven deep learning approaches, covering physics-driven loss functions, plug-and-play methods, generative models, and unrolled networks. We highlight domain-specific challenges such as real- and complex-valued building blocks of neural networks, and translational applications in MRI with linear and non-linear forward models. Finally, we discuss common issues and open challenges, and draw connections to the importance of physics-driven learning when combined with other downstream tasks in the medical imaging pipeline.

I. INTRODUCTION

The introduction frames computational MRI as recovering images or other quantities from measurements shaped by acquisition physics. It outlines the review’s coverage of linear and nonlinear inverse problems and physics-driven learning methods that use raw k-space data.

I. INTRODUCTION

  • After discretization, simplified image reconstruction uses a linear encoding model, while nonlinear models estimate broader unknown quantities that may include physical parameters.The nonlinear formulation distinguishes the unknown vector from an image alone; depending on the application, it can include motion fields or relaxation maps.Source: §I · pp. 3–4
  • Reconstructing images from subsampled or degraded k-space is a canonical computational MRI inverse problem that requires MRI encoding or image-domain knowledge.For some known imaging sequences, the signal model can be expressed analytically; otherwise, the Bloch equations are integrated directly.Source: §I · pp. 2–3
  • Classical inverse-problem approaches progressed from partial-Fourier symmetry to multicoil redundancies and compressed sensing that exploits MR-image compressibility.These approaches use k-space properties, coil information, or image structure to reconstruct from undersampled data.Source: §I · p. 3
  • The article reviews inverse problems in computational MRI and physics-driven deep learning methods that use raw k-space data.The introduction describes physics-driven methods as explicitly incorporating known forward imaging models into architectures or training to control measurement consistency.Source: §I · p. 4

II. CLASSICAL APPROACHES FOR COMPUTATIONAL MRI

Classical computational MRI reconstruction uses domain knowledge to recover images from undersampled measurements. The section describes parallel imaging and compressed sensing, then reviews iterative optimization approaches for regularized inverse problems.

II. CLASSICAL APPROACHES FOR COMPUTATIONAL MRI

  • At sub-Nyquist sampling, direct inverse Fourier reconstruction aliases, so image formation requires solving an inverse problem with additional domain knowledge.Parallel imaging uses receiver-coil redundancies; compressed sensing uses MR-image compressibility to reconstruct from subsampled k-space.Source: §II · p. 5
  • Although coil count sets a theoretical acceleration bound, dependent coil sensitivities and ill-conditioning can amplify noise and limit achievable rates.Compressed sensing addresses subsampled reconstruction with a regularized least-squares objective using a sparsifying transform.Source: §II · p. 5

A. Solving the linear inverse problem in classical computational MRI

  • Classical solvers include gradient descent for differentiable regularizers and proximal gradient methods for nonsmooth regularizers.Proximal gradient descent separates an image update from an intermediate-image proximal subproblem.Source: A. Solving the linear inverse problem… · pp. 5–6
  • Variable-splitting methods such as ADMM separate data-consistency, proximal-operator, and dual-update subproblems.A quadratic-penalty variant is also described, with the regularizer potentially implemented using machine-learning techniques.Source: A. Solving the linear inverse problem… · pp. 5–6

B. Solving the non-linear inverse problem in classical computational MRI

This section moves from the non-convex, model-sensitive optimization of nonlinear MRI inverse problems to physics-driven learning methods that incorporate acquisition physics into reconstruction. It reviews image- and k-space enhancement strategies and their limits, alongside classical approaches to nonlinear reconstruction.

B. Solving the non-linear inverse problem in classical computational MRI

III. PHYSICS-DRIVEN ML METHODS IN COMPUTATIONAL MRI

  • Image enhancement alone discards acquisition physics and cannot guarantee k-space consistency; the reviewed physics-driven methods span enhancement, plug-and-play and generative priors, and learned unrolled optimization.Source: §III · p. 7

A. Physics Information in Image/k-space Enhancement Methods

  • For Cartesian sampling, hard replacement of measured k-space lines after enhancement may cause artifacts, while a training-time consistency term cannot guarantee inference-time consistency, particularly for unseen pathologies.Source: A. Physics Information in… · p. 8
  • Nonlinear k-space interpolation learned from calibration data can work without large training databases, but may be limited at high acceleration when calibration data are insufficient.Source: A. Physics Information in… · p. 8
  • An alternative strategy applies scan-specific GRAPPA kernels before refining the reconstruction in both image and k-space domains.Source: A. Physics Information in… · p. 8

B. Plug-and-play Methods with Deep Denoisers

This section covers plug-and-play and generative-prior approaches to MRI reconstruction, including VAEs used as learned priors in Bayesian multi-coil reconstruction and to quantify acquisition uncertainty through sampling.

B. Plug-and-play Methods with Deep Denoisers

  • PnP replaces proximal denoisers with other denoising algorithms, while RED can explicitly tie its regularizer to a denoiser under certain conditions.These methods separate image modeling from the MRI acquisition physics.Source: B. Plug-and-play Methods with Deep… · p. 8
  • CNN denoisers are typically trained on reference images corrupted with added noise, learning a mapping from noisy to reference images.When references are unavailable, denoising can instead be trained from pairs of noisy images.Source: B. Plug-and-play Methods with Deep… · p. 9
  • RARE trains CNN denoisers on artifact-image pairs from split-measurement least-squares reconstructions, independently of the broader inverse problem.This allows one denoiser to be applied across undersampling rates or patterns; additional artifact information can help recognize characteristic aliasing.Source: B. Plug-and-play Methods with Deep… · p. 9

C. Generative Models

  • Supervised GAN reconstruction uses zero-filled images as generator input, then applies data consistency and adversarial loss to recover finer details.At inference, the generator produces the image and a data-consistency step follows; the discriminator supplies the adversarial loss during training.Source: C. Generative Models · pp. 10–11
  • Inverse GANs train on clean MRI references without physics knowledge, then optimize the latent vector and generator using undersampled k-space data at test time.This adapts the generator to the measurements and is not restricted to a sampling pattern.Source: C. Generative Models · pp. 11–12
  • CycleGAN methods minimize losses for cyclic consistency and GAN training, enforcing consistency in both image and k-space domains.The described cycle-consistency terms target the image and k-space domains, while the Wasserstein GAN loss is noted to improve training stability.Source: C. Generative Models · p. 12
  • VAEs trained on fully sampled MR image patches provide deep density priors that can be plugged into Bayesian multi-coil reconstruction.The VAE learns an image distribution from the patches before its prior is used in the reconstruction formulation.Source: C. Generative Models · pp. 12–13
  • For uncertainty quantification, a VAE encodes acquisition uncertainty and Monte Carlo sampling generates uncertainty maps from its learned distribution.Source: C. Generative Models · p. 13

D. Algorithm unrolling and unrolled networks

Algorithm unrolling turns iterative MRI reconstruction methods into finite-stage networks whose parameters can be learned end-to-end. These networks alternate learned regularization with data consistency tied to acquired k-space.

D. Algorithm unrolling and unrolled networks

  • Unrolling adapts iterative optimization for MRI into a fixed number of trainable stages, enabling end-to-end learning of reconstruction parameters.The approaches build on ADMM, gradient descent, proximal-gradient, primal-dual, and variable-splitting algorithms.Source: D. Algorithm unrolling and unrolled… · p. 13
  • Each stage combines a regularization-related processing step with data consistency that keeps image estimates faithful to acquired k-space.The data-consistency update depends on the optimization algorithm used for unrolling.Source: D. Algorithm unrolling and unrolled… · pp. 13–14
  • Variational Networks learn regularization gradients, while other schemes replace the gradient or proximal operator with a trainable CNN.The learned parameters may be shared across stages or vary by stage.Source: D. Algorithm unrolling and unrolled… · pp. 13, 15
  • Variable-splitting methods such as MoDL use a CNN-based regularizer and solve a data-consistency subproblem, often with conjugate gradients for multi-coil MRI.Single-coil systems may use direct matrix inversion; multi-coil inversion is computationally infeasible in the described setting.Source: D. Algorithm unrolling and unrolled… · p. 15
  • Proximal-gradient unrolling reduces data-consistency memory demands, especially for multi-coil data, but its performance lags the conjugate-gradient approach used in MoDL.The first fastMRI challenge reported similar performance for gradient-descent and variable-splitting algorithms; the second emphasized coil-sensitivity handling and regularization networks.Source: D. Algorithm unrolling and unrolled… · pp. 15–16
  • Learned unrolled schemes commonly use CNN or multiscale regularizers, including UNET, Down-Up Networks, and multilevel wavelet CNNs.Network parameters can be shared across reconstruction stages or varied between them.Source: D. Algorithm unrolling and unrolled… · p. 16

1) Training unrolled networks:

Unrolled networks are commonly trained end-to-end with supervised reference data, but self-supervised and unsupervised alternatives address settings where fully sampled references are unavailable. Single-dataset training also raises overfitting concerns.

1) Training unrolled networks:

  • For end-to-end training of unrolled networks, supervised learning commonly uses paired input and reference data, with fully-sampled measurements or ground-truth images as references; losses compare network output and reference in image, k-space, or mixed domains.Source: 1) Training unrolled networks · p. 16
  • Self-supervised learning can train without fully sampled data by generating training targets from undersampled measurements, as in SSDU.SSDU partitions acquired measurements into disjoint sets: one enforces network data consistency, while the unseen set defines a k-space loss.Source: 1) Training unrolled networks · pp. 16–17
  • SSDU measures discrepancy between network-output measurements and acquired measurements at unseen locations, then uses all acquired data for test reconstruction.The network output is transformed back to k-space using the encoding operator.Source: 1) Training unrolled networks · p. 17
  • Stein’s unbiased risk estimate of mean square error supports unsupervised MRI training, with an ensemble-based loss shown to be unbiased.The cited setup uses images acquired with different undersampling operators.Source: 1) Training unrolled networks · p. 17
  • Single-dataset unrolled-network training may overfit when trainable parameters outnumber pixels or k-space measurements; zero-shot self-supervision addresses this challenge.A third measurement partition supplies a self-validation loss that rises when overfitting is observed, guiding the stopping epoch.Source: 1) Training unrolled networks · p. 17

2) Memory challenges of unrolled networks and deep equilibrium networks:

Unrolled networks can demand substantial memory and transfer resources, while deep equilibrium networks reduce training storage by modeling the infinite-iteration limit. The alternatives trade memory and convergence behavior against inference-time and training-cost constraints.

2) Memory challenges of unrolled networks and deep equilibrium networks:

  • Unrolled networks have large memory footprints because training stores CNNs and data-consistency operations across iterations, challenging large-scale and multidimensional datasets.Deeper networks tend to improve performance, intensifying the memory challenge.Source: 2) Memory challenges of unrolled… · p. 17
  • Memory-efficient learning stores intermediate iteration outputs in host memory and computes gradients from those outputs, conceptually permitting as many unrolling steps as desired.The approach adds data-transfer costs between host and GPU memory.Source: 2) Memory challenges of unrolled… · pp. 17–18
  • Deep equilibrium networks characterize the limit of a shared-stage unrolled network as iterations approach infinity, corresponding to a fixed-point solution when the limit exists.Only one stage must be stored during training, reducing memory use.Source: 2) Memory challenges of unrolled… · p. 18
  • Deep equilibrium networks have better-behaved convergence across inference iteration counts, but lack fixed inference time and require costly Jacobian inversion during training.Quasi-Newton approximations to the inverse Jacobian have reduced computation cost.Source: 2) Memory challenges of unrolled… · p. 18

IV. STATE-OF-THE-ART IN MRI PRACTICE AND DOMAIN-SPECIFIC CHALLENGES

MRI networks can represent complex data either as stacked real-valued channels or with complex-valued operations, each with distinct trade-offs. Convolution and activation choices must account for how operations affect phase and magnitude.

A. Real vs complex building blocks

  • Stacked real/imaginary channels permit real-valued operations but lose the complex relationship; complex operations preserve it, while some require twice the trainable parameters and multiplications.Complex convolutions learn separate real and imaginary filter kernels.Source: A. Real vs complex building blocks · p. 18

1) Convolution:

  • Complex convolution maps input feature channels to output channels using complex-valued kernels, combining real and imaginary component convolutions.The paper also distinguishes real-valued convolution, where K = R, from complex convolution, where K = C.Source: 1) Convolution · p. 19

2) Activation:

  • Activation choices affect phase and magnitude: separable ReLUs ignore real–imaginary correlation and discard phase information, while alternatives can retain phase.The cited work reports complex networks outperforming real-valued networks in water-fat and phase-contrast imaging, with separable ReLUs performing best.Source: 2) Activation · pp. 19–20
  • Activation selection is application-dependent: the complex cardioid acts on phase, whereas ModReLU fixes phase and alters magnitude; complex cardioid performed best for MR fingerprinting.The cardioid is described as a generalization of ReLU to the complex plane, and a trainable bias can be added.Source: 2) Activation · p. 20

3) Normalization:

Normalization may stabilize training but can introduce image-restoration artifacts, and complex pooling modifies the maximum operation. The linear-model examples illustrate domain-shift risks and reconstruction approaches that incorporate data consistency.

3) Normalization:

  • Normalization layers can support faster, more stable training, but their task-dependent choice may produce unwanted artifacts in image restoration.Complex-valued normalization estimates normalization using a covariance matrix [70].Source: 3) Normalization · p. 20

4) Pooling:

  • Complex-valued pooling replaces a maximum operation that does not exist for complex images with selection based, for example, on maximum magnitude.Pooling also downsamples spatial resolution and introduces approximate invariance to small translations.Source: 4) Pooling · p. 20

B. Canonical MRI reconstruction with the linear forward model

  • For non-Cartesian sampling, physics-driven methods include data consistency and expressive regularization networks, and pathologies need not appear in training if sufficient k-space data guides reconstruction.The cited approaches also include jointly learned non-Cartesian trajectories and reconstruction.Source: B. Canonical MRI reconstruction with… · p. 21

4) Pooling:

  • At acceleration 4, Down-Up networks generalized across knee and neuro data, but at acceleration 8, knee-trained networks applied to neuro data reconstructed ventricles resembling knee structures.The described study used fastMRI knee and neuro datasets.Source: B. Canonical MRI reconstruction with… · p. 22

B. Canonical MRI reconstruction with the linear forward model

  • CINENet combines data-consistency layers with complex-valued building blocks and a UNet architecture for dynamic 3D+t data.Its complex-valued blocks include convolutions, activations, pooling, and normalization; convolutions are split into 3D spatial and 1D temporal operations.Source: Figure 5 · p. 22, B. Canonical MRI reconstruction with… · p. 22
  • Unrolled networks trained on prospectively accelerated late gadolinium enhancement acquisitions without fully sampled references improved on clinically used compressed sensing and allowed twice-as-fast acceleration.The acquisitions cannot be fully sampled because of contrast-related scan-time constraints; self-supervised training is also reported to outperform current clinically used approaches.Source: B. Canonical MRI reconstruction with… · p. 23

C. Inverse problems in MRI with non-linear forward models

For nonlinear MRI inverse problems, deep learning maps acquired data to unknown parameters through supervised or self-supervised training. The review surveys applications in quantitative mapping, susceptibility, flow and perfusion, and motion.

C. Inverse problems in MRI with non-linear forward models

  • Deep networks map acquired data to unknown parameters such as magnetization and relaxation maps, using either supervised or self-supervised training.The review frames these models as addressing computationally demanding nonlinear MRI inverse problems.Source: C. Inverse problems in MRI with… · pp. 23–24

1) Relaxivity mapping:

  • Relaxivity mapping seeks multiple parametric maps from one scan; model-based reconstruction avoids reconstructing individual images along the relaxivity curve but has prolonged reconstruction times.Deep learning has been proposed for fast inference, shifting time-demanding workloads to offline training.Source: 1) Relaxivity mapping · p. 24

2) Susceptibility mapping:

  • Physics-driven susceptibility mapping has incorporated the dipole inversion model in training losses, unrolled gradient descent for nonlinear dipole inversion, and scan-specific fine-tuning using the physics model.The cited scan-specific approach resembles the loss in Eq. (34) without additional regularizers.Source: 2) Susceptibility mapping · p. 24

3) Flow, perfusion and contrast-enhanced MRI:

  • Flow imaging requires considerable acquisition time, while perfusion acquisitions can have low resolution or low signal-to-noise ratio; proposed methods incorporate physical flow equations or kinetic models.Examples include enforcing fidelity to a known ground truth, solving fluid-flow partial differential equations through automatic differentiation, and incorporating kinetic models into losses.Source: 3) Flow, perfusion and… · p. 24

4) Motion:

  • Motion-aware methods estimate motion fields and embed them in data consistency or the forward model; one unrolled approach jointly trains reconstruction and motion-estimation losses.A coronary MRI approach estimates diffeomorphic fields from zero-filled images and embeds them in a conjugate-gradient data-consistency layer.Source: 4) Motion · p. 25
  • Embedding motion-estimation UNets in an unrolled network's data-consistency layers allows both reconstruction and motion estimation to improve as the motion networks rely on earlier reconstructions.Source: 4) Motion · p. 25

V. DISCUSSION

The discussion highlights unresolved clinical, generalization, and acquisition concerns for computational MRI. It emphasizes that realistic-looking reconstructions may be inconsistent with measurements and that retrospective evaluation may omit effects encountered in prospective scanning.

A. Issues and open problems

  • Deep learning can generate realistic-looking images with missing or hallucinated features inconsistent with measurements when acceleration exceeds the encoding capability of the multi-element system.The discussion contrasts these outputs with characteristic artifacts from classic regularizers, which are easier for end-users to recognize as technical artifacts.Source: A. Issues and open problems · p. 26
  • The discussion proposes shifting assessment from qualitative image quality toward clinical outcomes.It also identifies open questions on basic science and translation.Source: A. Issues and open problems · pp. 25–26
  • Whether deep learning improves patients' diagnostic quality over handcrafted regularizers remains an open question, and clinical studies are slow and costly.Imaging exams may not provide true ground truth and can require follow-up with pathology or surgery departments.Source: A. Issues and open problems · p. 26
  • Scanner generalization tests can miss sensitivity to seemingly trivial data changes, including whether readout-direction oversampling is retained.The fastMRI 2020 challenge included a different-manufacturer scanner deployment track; winning models performed well, but some approaches were negatively affected by such data modifications.Source: A. Issues and open problems · p. 27
  • Retrospective undersampling of fully sampled data does not capture all prospective acquisition effects, including spin-history and gradient or RF-hardware effects, which can cause issues on real scanners.The discussion characterizes this as a general issue for computational imaging methods developed retrospectively, not a problem unique to deep learning.Source: A. Issues and open problems · p. 27
  • Using DICOM images instead of raw k-space data may yield optimistic experimental results, while real-world unprocessed data performs much worse and can bias reported state-of-the-art results.Source: A. Issues and open problems · p. 27
  • Different sampling patterns can yield different reconstruction results, making acceleration-factor labels alone insufficient to characterize performance.The discussion raises how to identify potential mis-reconstructions and quantify uncertainty regarding the sampling pattern.Source: A. Issues and open problems · p. 27
  • Acceleration rates require careful interpretation: Cartesian undersampling is based on sampled versus total k-space lines, whereas non-Cartesian effective undersampling also depends on readout oversampling.Source: A. Issues and open problems · p. 27

B. Domain-specific knowledge in post-processing and multi-task imaging

This section reviews physics-driven knowledge in post-processing and multi-task imaging, including joint MRI reconstruction and downstream tasks. It concludes that physics-driven deep learning is current state of the art for computational MRI and is beginning to enter translational and clinical settings.

B. Domain-specific knowledge in post-processing and multi-task imaging

  • Joint motion detection, correction, and segmentation can detect motion directly in k-space, influencing the data-consistency layer.A bidirectional recurrent network is also used.Source: B. Domain-specific knowledge in… · p. 28
  • On UK Biobank data, joint reconstruction and segmentation training outperformed sequential training of the networks.Source: B. Domain-specific knowledge in… · p. 28
  • A unified network shares an encoder between reconstruction and segmentation while task-specific decoders adapt to each task.Evaluation and k-space simulation used the MRBrainS segmentation challenge dataset, and the results suggest high-quality segmentation benefits from this multi-task architecture.Source: B. Domain-specific knowledge in… · p. 28
  • FR-Net combines FISTA-based reconstruction and UNet segmentation losses, with the trade-off between the two terms key to high-quality results.Source: B. Domain-specific knowledge in… · p. 28
  • Whether downstream segmentation needs intermediate reconstructed images or can be obtained directly in k-space, as proposed in, remains open.Source: B. Domain-specific knowledge in… · p. 28
  • Post-processing can incorporate domain knowledge through differentiable registration similarity measures and segmentation priors on object shape, topology, or location.Segmentation approaches include regularization losses and deforming a topologically consistent candidate shape to match reference data [107].Source: B. Domain-specific knowledge in… · pp. 28–29

VI. CONCLUSION

  • Physics-driven methods span loss functions, plug-and-play techniques, generative models, and unrolled networks for linear and nonlinear MRI inverse problems.The conclusion identifies these methods as current state of the art for computational MRI.Source: §VI · p. 29
  • These methods are beginning to enter translational and clinical settings and may alter downstream tasks in the medical imaging pipeline.The conclusion notes opportunities for further technical developments and applications.Source: §VI · p. 29
Loading 2203.12215v3…