PaperlayerView exact revision on arXiv

Source-linked AI summary

Conversion Between CT and MRI Images Using Diffusion and Score-Matching Models

Qing Lyu, Ge Wang

arXiv:2209.12104v2eess.IVcs.CVphysics.med-ph

TL;DR

Separate MRI and CT imaging is costly and can leave scans misaligned, motivating computational conversion. This paper adapts diffusion and score-matching models for MRI-conditioned CT synthesis and compares four sampling strategies with CNN and GAN models. The authors report realistic synthetic CT images, favorable comparisons with CNN and GAN results, and improved outputs from Monte Carlo averaging.

  • Problem

    Separate MRI and CT scans are costly and can be misaligned, motivating computational conversion between the complementary modalities.

    Source: §I · p. 1

  • Method

    The paper adapts diffusion and score-matching models for MRI-to-CT synthesis and evaluates four sampling strategies against CNN and GAN models.

    Source: §I · p. 1, §V · p. 8

  • Results

    The four strategies generated realistic CT images, their results compared favorably with CNN and GAN models, and Monte Carlo averaging improved the outputs.

    Source: §V · pp. 8–9

  • Takeaways & Limitations

    The authors conclude that diffusion and score-matching models are promising for image synthesis between CT and MRI, with uncertainty also quantified.

    Source: §V · p. 9

    Existing unsupervised diffusion methods have conditioning constraints described as too weak or too strong for CT-MRI synthesis, motivating work to balance them.

    Source: §V · p. 9

Abstract

from arXiv · show

MRI and CT are most widely used medical imaging modalities. It is often necessary to acquire multi-modality images for diagnosis and treatment such as radiotherapy planning. However, multi-modality imaging is not only costly but also introduces misalignment between MRI and CT images. To address this challenge, computational conversion is a viable approach between MRI and CT images, especially from MRI to CT images. In this paper, we propose to use an emerging deep learning framework called diffusion and score-matching models in this context. Specifically, we adapt denoising diffusion probabilistic and score-matching models, use four different sampling strategies, and compare their performance metrics with that using a convolutional neural network and a generative adversarial network model. Our results show that the diffusion and score-matching models generate better synthetic CT images than the CNN and GAN models. Furthermore, we investigate the uncertainties associated with the diffusion and score-matching networks using the Monte-Carlo method, and improve the results by averaging their Monte-Carlo outputs. Our study suggests that diffusion and score-matching models are powerful to generate high quality images conditioned on an image obtained using a complementary imaging modality, analytically rigorous with clear explainability, and highly competitive with CNNs and GANs for image synthesis.

I. INTRODUCTION

MRI and CT provide complementary information, but separate scans are costly and can be misaligned, motivating computational image synthesis. The section surveys deep-learning synthesis work and introduces diffusion and score-matching models for MRI-to-CT conversion.

I. INTRODUCTION

  • Separate MRI and CT scans provide complementary clinical information but are costly and can be nonrigidly misaligned, motivating computational synthesis as an alternative.For radiotherapy, CT provides electron density for planning while MRI outlines tumors and soft tissues.Source: §I · p. 1

A. Deep medical image synthesis

  • Prior cross-modality synthesis work largely uses CNNs and GANs, including MRI-to-CT systems and models for other modality pairs.Reported examples include fully convolutional networks and cascaded or conditional GANs for MRI-to-CT, alongside PET synthesis systems.Source: §I · p. 1, A. Deep medical image synthesis · pp. 1–2

I. INTRODUCTION

  • The paper applies DDPM and SDE-based models to CT/MRI conversion, emphasizes MRI-to-CT mapping, and compares them with CNN and GAN models.It also estimates uncertainty from Monte Carlo samples and reports improved results by averaging those samples.Source: §I · p. 1

B. Diffusion and score-matching model

  • Diffusion and score-matching models gradually add noise and then denoise, and are presented as analytically principled generative frameworks.Representative frameworks named here are DDPM, NCSN, and SDE.Source: B. Diffusion and score-matching model · p. 2

1) Denoising diffusion probabilistic models:

DDPM gradually corrupts an image with Gaussian noise and trains a reverse process to recover it. Its training objective uses a variational lower bound, and the paper presents the conditional model's training and sampling procedures.

1) Denoising diffusion probabilistic models:

  • DDPM gradually corrupts an image through multiple steps, with incremental Gaussian noise controlled by β_t, until the terminal distribution approaches isotropic Gaussian noise.The passage describes the transitions as a Markovian process and notes the Gaussian limit as T approaches infinity.Source: 1) Denoising diffusion probabilistic… · p. 2
  • A neural network approximates each reverse diffusion step by estimating its mean and covariance, with the reverse step tractable when conditioned on x_t and x_0.The passage also gives a reparameterization of x_0 using the noisy sample and added noise.Source: 1) Denoising diffusion probabilistic… · p. 2
  • The noise-estimation network is trained using a variational lower bound whose KL terms compare reverse-step distributions; the paper lists conditional DDPM training and sampling procedures.The terminal loss term L_T is constant and can be ignored because it has no learnable parameters.Source: 1) Denoising diffusion probabilistic… · pp. 2–3, Table I · p. 3

2) Noise conditioned score network:

The section describes score-based sampling with Langevin dynamics and training a network to estimate data-distribution scores. It also presents noise-perturbed score estimation across multiple scales.

2) Noise conditioned score network:

  • Langevin dynamics uses the score function and random noise to sample from a probability density, with the stated distributional limit as the step size tends to zero and steps increase.The passage specifies an initial value drawn from a prior distribution.Source: 2) Noise conditioned score network · p. 3
  • A neural network is trained to estimate the score, ∇_x log p_t(x), used in the sampling process.The section identifies score matching as the training approach.Source: 2) Noise conditioned score network · p. 3
  • Because the original score is difficult to obtain, NCSN perturbs the data distribution with Gaussian noise at increasing scales and estimates scores for the perturbed distributions.The noise-scale objective includes a weighting factor λ(σ_t).Source: 2) Noise conditioned score network · p. 3

3) Stochastic differential equation:

The SDE framework represents diffusion as a continuous process that generalizes discrete DDPM and NCSN. Its reverse-time process uses the data score, which a neural network is trained to estimate.

3) Stochastic differential equation:

  • The SDE framework continuously transforms the original data distribution toward a Gaussian prior, generalizing DDPM and NCSN's discrete diffusion processes.The terminal prior is described as unstructured and without information from the original distribution.Source: 3) Stochastic differential equation · p. 3
  • The forward SDE uses drift and diffusion coefficients, while its reverse-time form includes the data-distribution score and reversed-time Brownian motion.The passage identifies f and g as the drift and diffusion coefficients, respectively.Source: 3) Stochastic differential equation · p. 3
  • A neural network estimates the score for the reverse process, and the training objective uses a continuous score-matching formulation with a positive weighting function.The formulation samples time uniformly over [0, T] and uses a conditional transition score in place of the original score.Source: 3) Stochastic differential equation · p. 3

A. Data

The study uses co-registered T2w MRI–CT pairs from 19 patients and adapts conditional DDPM to map T2w images to CT.

A. Data

  • 17 patients supplied 1,416 image pairs for training, while two patients supplied 135 pairs for testing.The Gold Atlas male pelvis dataset contains co-registered T2w MRI and CT images from 19 patients.Source: A. Data · pp. 3–4

B. Conditional DDPM

  • Conditional DDPM is extended from within-modality photographs to cross-modality CT synthesis conditioned on T2w MRI.Its objective is defined over co-registered CT and T2w MRI pairs.Source: B. Conditional DDPM · p. 4
  • Sampling starts from Gaussian noise and reverses a conditional Markov process; the implementation uses 1,000 diffusion and sampling steps with a UNet denoiser.The reverse transition is specified as a Gaussian conditional on the current state, original image, and T2w MRI.Source: B. Conditional DDPM · p. 4

C. Conditional SDE

The conditional SDE model supervises diffusion with T2w MRI and samples through reverse-time dynamics using three numerical strategies.

C. Conditional SDE

  • T2w MRI is included during training to supervise both forward and backward diffusion, using the variance-exploding SDE setting with f = 0 and g = σt.The paper contrasts this supervised conditioning with classifier-guidance approaches that incorporate conditions during sampling.Source: C. Conditional SDE · p. 4
  • Figure 1 presents intermediate reverse-process results for DDPM, ODE, EM, and PC, with a T2w MR condition and CT ground truth in the first row.The figure passage identifies the methods and row contents but does not state a comparative winner.Source: Figure 1 · p. 5
  • Sampling begins from a Gaussian prior and solves the reverse-time SDE using Euler–Maruyama, Prediction-Correction, or probability-flow ODE methods.When σ(T) is large, the prior mean is approximated as zero.Source: C. Conditional SDE · p. 5

1) Euler-Maruyama method:

The SDE sampling section describes Euler–Maruyama discretization and a Prediction-Correction alternative that alternates prediction with score-based correction.

1) Euler-Maruyama method:

  • Euler–Maruyama solves the reverse-time SDE by replacing dt with a small increment and Brownian noise with Gaussian noise.The increment noise is drawn from a zero-mean Gaussian with covariance Δt · I.Source: 1) Euler-Maruyama method · p. 5

2) Prediction-Correction method:

  • Prediction-Correction alternates a reverse-SDE predictor with a score-based MCMC corrector, such as annealed Langevin dynamics.The Langevin step size uses a signal-to-noise ratio and Gaussian noise.Source: 2) Prediction-Correction method · pp. 5–6
  • Figure 2 shows repeated results conditioned on the same MR image, with ten-sample averages and pixel-wise standard-deviation maps in the bottom row.Images are scaled to [0, 1], while standard-deviation maps are scaled to [0, 0.5].Source: Figure 2 · p. 6

3) Probability flow ODE method:

The probability-flow ODE shares the SDE’s marginal probability trajectory and is solved backward from a prior to generate samples.

3) Probability flow ODE method:

  • For an SDE, the associated probability-flow ODE has the same marginal probability-density trajectory as the SDE.The paper calls this sampling approach ODE for simplicity.Source: 3) Probability flow ODE method · p. 6
  • ODE sampling starts from the prior, integrates in reverse time, and obtains a sample from p0 using an order 5(4) Explicit Runge-Kutta solver.The reverse-time ODE sampling is described as equivalent to solving the reverse-time SDE.Source: 3) Probability flow ODE method · p. 6
  • All compared sampling methods use 1,000 total steps, with PC divided into 500 prediction and 500 correction steps.The procedures are listed in Table II.Source: 3) Probability flow ODE method · p. 6

D. Other methods for comparison

The section compares image-synthesis methods and evaluates quality and uncertainty. Averaging Monte Carlo outputs improves several sampling methods’ SSIM and PSNR, while EM and PC show lower uncertainty than DDPM and ODE.

D. Other methods for comparison

E. Implementation details

  • Ten outputs per target CT were averaged to form Monte Carlo results, and their standard-deviation maps were used to reveal model uncertainty.SSIM and PSNR were used to evaluate image quality.Source: E. Implementation details · p. 7, §IV · p. 7

IV. RESULTS

  • Image quality was evaluated with structural similarity index measure (SSIM) and peak signal-to-noise ratio (PSNR).Source: §IV · p. 7

A. Diffusion and score-matching results

  • DDPM, EM, PC, and ODE progressively remove noise from Gaussian noise to generate a desirable CT image conditioned on the associated T2w MRI.The intermediate reverse-process results are compared in Fig. 1.Source: A. Diffusion and score-matching results · p. 7

B. Model uncertainty estimation

  • Averaging ten Monte Carlo samples improved ODE, EM, and PC SSIM/PSNR over their individual outputs; EM/PC also had lower uncertainty than DDPM/ODE.The uncertainty comparison is supported by both standard-deviation maps and average model-uncertainty scores.Source: B. Model uncertainty estimation · pp. 7–8

C. Comparison with CNN and GAN

Diffusion and score-matching methods produce detailed synthetic CT images, with Monte Carlo averaging improving results across sampling strategies. Their quality, speed, and conditioning limitations shape the practical trade-offs discussed.

C. Comparison with CNN and GAN

  • Diffusion and score-matching results retain faithful detail, unlike CNN over-smoothing and WGAN-GP artifacts; Monte Carlo averages improve fidelity, SSIM, and PSNR over individual samples.The averaged DDPM-M, ODE-M, EM-M, and PC-M results also score higher in SSIM and PSNR than their corresponding individual methods.Source: C. Comparison with CNN and GAN · p. 8

V. DISCUSSIONS AND CONCLUSION

  • Across DDPM, ODE, EM, and PC sampling, ODE has inferior quality but is significantly faster, while PC is slowest; the authors recommend EM as a good SDE-based choice.The authors advise balancing speed and quality for the application.Source: §V · pp. 8–9, C. Comparison with CNN and GAN · p. 8
  • DDPM has greater model uncertainty than SDE-based methods; the authors infer discrete-time training may contribute and hypothesize more diffusion and reverse steps could reduce it.The proposed explanation and remedy are presented as an inference and assumption, respectively.Source: §V · pp. 8–9
  • Diffusion and score-matching sampling relies on a long Markov chain and is relatively slow, with a remaining speed gap versus CNN and GAN inference.The authors identify faster sampling as a future direction.Source: §V · p. 9
  • For unsupervised diffusion methods, cited conditioning approaches are either too weak to reach specific structures or too strong for CT-MRI similarity, motivating a balance.The authors identify balancing these constraints as future work for high-quality CT-MRI synthesis.Source: §V · p. 9
Loading 2209.12104v2…