Source-linked AI summary
Synthetic CT Generation from MRI using 3D Transformer-based Denoising Diffusion Model
Shaoyan Pan, Elham Abouei, Jacob Wynne, Tonghe Wang, Richard L. J. Qiu, Yuheng Li, Chih-Wei Chang, Junbo Peng, Justin Roper, Pretesh Patel, David S. Yu, Hui Mao, Xiaofeng Yang
TL;DR
Radiotherapy planning benefits from both MRI and CT, but acquiring and registering both can add uncertainty, radiation exposure, cost, and discomfort. The paper proposes MC-DDPM, a 3D MRI-conditioned diffusion model with a shifted-window transformer, and reports high-quality brain and prostate sCT results, alongside a computational-efficiency cost.
Problem
MRI and CT provide complementary information for radiotherapy planning, but using both can involve co-registration uncertainty, radiation exposure, extra costs, and patient discomfort.
Method
MC-DDPM uses a 3D shifted-window transformer network to learn a diffusion process that generates synthetic CT from MRI inputs.
Results
MC-DDPM generated brain sCTs with MAE 43.317±4.104 HU, PSNR 27.046±0.817 dB, SSIM 0.965±0.005, and NCC 0.983±0.004, with statistical improvement over competing methods.
Takeaways & Limitations
MC-DDPM generates high-quality sCT images using only MRI inputs, supporting MRI-to-CT synthesis without requiring CT scans in the generation process.
Takeaways & Limitations
The reported performance comes at the cost of computational efficiency because MC-DDPM relies on a long diffusion-step Markov chain.
Abstract
from arXiv · showhide
Magnetic resonance imaging (MRI)-based synthetic computed tomography (sCT) simplifies radiation therapy treatment planning by eliminating the need for CT simulation and error-prone image registration, ultimately reducing patient radiation dose and setup uncertainty. We propose an MRI-to-CT transformer-based denoising diffusion probabilistic model (MC-DDPM) to transform MRI into high-quality sCT to facilitate radiation treatment planning. MC-DDPM implements diffusion processes with a shifted-window transformer network to generate sCT from MRI. The proposed model consists of two processes: a forward process which adds Gaussian noise to real CT scans, and a reverse process in which a shifted-window transformer V-net (Swin-Vnet) denoises the noisy CT scans conditioned on the MRI from the same patient to produce noise-free CT scans. With an optimally trained Swin-Vnet, the reverse diffusion process was used to generate sCT scans matching MRI anatomy. We evaluated the proposed method by generating sCT from MRI on a brain dataset and a prostate dataset. Qualitative evaluation was performed using the mean absolute error (MAE) of Hounsfield unit (HU), peak signal to noise ratio (PSNR), multi-scale Structure Similarity index (MS-SSIM) and normalized cross correlation (NCC) indexes between ground truth CTs and sCTs. MC-DDPM generated brain sCTs with state-of-the-art quantitative results with MAE 43.317 HU, PSNR 27.046 dB, SSIM 0.965, and NCC 0.983. For the prostate dataset, MC-DDPM achieved MAE 59.953 HU, PSNR 26.920 dB, SSIM 0.849, and NCC 0.948. In conclusion, we have developed and validated a novel approach for generating CT images from routine MRIs using a transformer-based DDPM. This model effectively captures the complex relationship between CT and MRI images, allowing for robust and high-quality synthetic CT (sCT) images to be generated in minutes.
1. Introduction
MRI offers strong soft-tissue and anatomical information, while CT provides geometric accuracy and electron-density data needed for radiotherapy planning. MRI-based sCT generation can reduce registration uncertainty and imaging burdens, but existing methods face algorithmic limitations that motivate MC-DDPM.
- MRI provides superior soft-tissue contrast and anatomical information, whereas CT supplies geometric accuracy and electron-density information for radiotherapy dose calculation.
- MRI-based sCT could support radiotherapy planning while reducing MRI–CT co-registration uncertainty, radiation exposure, extra imaging costs, and patient discomfort.
- Existing sCT methods depend on registration, segmentation, or multiple MRI sequences, creating sensitivity to algorithm accuracy, tissue ambiguity, scanning time, motion artifacts, or discomfort.
- GAN-based sCT methods can generate realistic images but suffer from unstable training, mode collapse, and output homogeneity associated with adversarial training.
- MC-DDPM introduces a 3D MRI-conditioned diffusion approach using sequential denoising steps and a shifted-window transformer to generate high-quality sCT with reduced artifacts.
- The proposed diffusion process is presented as improving image quality and training stability, while its long Markov chain makes MC-DDPM computationally expensive.
2. Method
MC-DDPM converts MRI into patient-matched synthetic CT through forward Gaussian diffusion and MRI-conditioned reverse denoising with a 3D Swin-Vnet transformer.
- Diffusion framework: MC-DDPM converts a patient’s MRI into a synthetic CT through forward diffusion and a reverse process that removes Gaussian noise using Swin-Vnet.The reverse process is conditioned on the corresponding MR scan and estimates less noisy CT images at successive timesteps.
- Diffusion framework: The diffusion process assumes that adding small amounts of noise over many timesteps transforms real CT into Gaussian noise, which reverse denoising can recover.The forward process uses a Markov chain with predetermined timestep variances, while the reverse process predicts denoised CT images.
- Generation procedure: The stochastic reverse process generates final sCT scans using Monte Carlo-based generation with five repeated runs.Randomness enters through the noise sample used during generation.
- Swin-Vnet architecture: The reverse network receives noisy CT and same-patient MRI, using a three-dimensional U-shaped encoder-decoder to estimate noise and variance coefficients.These estimates are used to calculate the less noisy CT image at the preceding timestep.
- Swin-Vnet architecture: Swin-Vnet combines convolutional blocks with window and shifted-window self-attention to learn local and global three-dimensional features.Encoder-decoder skip connections convey high-resolution information, while residual connections improve stability and reduce overfitting.
- Timestep conditioning: The architecture embeds diffusion timesteps with sinusoidal features so each network block can condition denoising on the current timestep.The timestep embedding has maximum period 10^6 and feature dimension 128.
3. Data Acquisition and Preprocessing
The study used paired MR-CT datasets from institutional brain and prostate cohorts, with registration, cropping, normalization, and patch-based processing before model training and inference.
- 3.A Institutional brain dataset: 36 brain patients were split into 28 training, 2 validation, and 6 testing cases, while scans were cropped to 192x192x96 voxel pairs.Brain scans used 1x1x1 mm3 voxels and central cropping focused on the cerebrospinal fluid region.
- 3. Data Acquisition and Preprocessing: CT intensities were clipped to [-1024, 1650] and jointly normalized to [-1, 1], while MRI intensities were independently normalized to [-1, 1].The brain model used AdamW for 500 epochs; the prostate model used AdamW for 800 epochs.
- 3. Data Acquisition and Preprocessing: Brain and prostate MRIs were paired with corresponding CT scans after clinical registration using Velocity AI 3.2.1.The brain dataset used Siemens Avanto MRI and Siemens SOMATOM CT scanners; the prostate dataset used GE Signa MRI and Siemens SOMATOM CT scanners.
- 3. Data Acquisition and Preprocessing: Training used randomly selected 128×128×4 prostate patches and 64x64x4 brain patches, with sliding-window inference and 50% overlap for brain scans.Gaussian weighting was applied at window edges, and no augmentation was used during inference.
4. Implementation and performance evaluation
MC-DDPM was evaluated against GAN- and DDPM-based MRI-to-CT methods using image-quality metrics, paired statistical tests, and qualitative comparisons across brain and prostate data.
- 4.A Implementation details: All experiments ran in PyTorch on Python 3.8.11 using a Windows 11 workstation with a single NVIDIA RTX 6000 GPU with 48GB memory.
- 4.B Synthetic CT Evaluation: MAE, PSNR, MS-SSIM, and NCC quantified absolute intensity difference, signal quality, visual similarity, and image correlation against clinical CT.Higher PSNR, MS-SSIM, and NCC indicate better sCT quality.
- 4. Implementation and performance evaluation: Brain and prostate qualitative figures show ground-truth CT or matched MRI inputs alongside MC-DDPM and competing sCT outputs with difference maps.The brain visualization uses selected columns for outputs and difference maps, while the prostate visualization presents outputs and corresponding difference maps in paired columns.
5. Results
MC-DDPM produced the strongest reported quantitative sCT results across brain and prostate datasets, outperforming competing GAN and DDPM approaches on most evaluated metrics.
- 5.A Quantitative result using the brain dataset: 43.317±4.104 HU MAE, 27.046±0.817 dB PSNR, 0.965±0.005 SSIM, and 0.983±0.004 NCC were best among brain methods.The result is reported in the brain comparison against MT-GAN, MT-CGAN, 2D-IDDPM, and 3D-DDPM.
- 5. Results: 2D-IDDPM outperformed GAN-based methods quantitatively, whereas 3D-DDPM showed worse results under the investigated settings.The authors suggest that timestep requirements may explain the weaker 3D-DDPM performance.
- 5.A Quantitative result using the brain dataset: MC-DDPM significantly improved brain MAE, PSNR, and NCC over 2D-IDDPM, but not SSIM, while improving all metrics over MT-GAN, MT-CGAN, and 3D-DDPM.The reported significance threshold was p<0.05.
- 5.B Quantitative result using the prostate dataset: 59.953±12.462 HU MAE, 26.920±2.429 dB PSNR, 0.849±0.041 SSIM, and 0.948±0.018 NCC were best among prostate methods.The prostate table reports MC-DDPM alongside MT-GAN, MT-CGAN, 2D-IDDPM, and 3D-DDPM.
- 5.B Quantitative result using the prostate dataset: MC-DDPM significantly improved all metrics over GAN-based methods and 3D-DDPM on the prostate dataset.Compared with 2D-IDDPM, significant prostate improvements were reported for MAE and SSIM.
6. Discussion
MC-DDPM combines diffusion modeling with a 3D Swin-transformer network to generate MRI-conditioned synthetic CT images. It achieves state-of-the-art results on institutional brain and prostate datasets, while its high image quality is accompanied by substantially slower generation than GAN-based methods.
- Method: The model combines diffusion models with Swin-transformer layers, using convolutional layers for local information and Swin attention for lower-resolution global characteristics.This design is described as the first diffusion model for MRI-to-sCT synthesis using a 3D Swin-transformer-based network.
- Method: MC-DDPM uses forward diffusion to add Gaussian noise to CT images and reverse diffusion with a Swin-Vnet conditioned on MRI to iteratively produce sCT.The Swin-Vnet estimates accumulated noise and a variance interpolation coefficient within a U-shaped encoder-decoder architecture.
- Results: MC-DDPM achieved state-of-the-art results on brain and prostate datasets, with statistically significant improvements over competing methods.Brain performance was MAE 43.317±4.104 HU, PSNR 27.046±0.817 dB, SSIM 0.965±0.005, and NCC 0.983±0.004; prostate performance was MAE 59.953±12.462 HU, PSNR 26.920±2.429 dB, SSIM 0.849±0.041, and NCC 0.948±0.018.
- Results: MC-DDPM outperformed GAN-based methods across all metrics and improved MAE and SSIM compared with 2D-IDDPM.The discussion also reports greater stability and lower sensitivity to hyper-parameter settings than GAN-based methods.
- Limitations: The authors plan efficiency improvements, deterministic diffusion, broader modality coverage, and more comprehensive validation in future work.They specifically identify reducing timesteps, accelerating sampling, and eliminating multiple Monte Carlo runs as future directions.
7. Conclusion
The study presents MC-DDPM, a 3D MRI-to-CT denoising diffusion model using a shifted-window transformer to generate synthetic CT scans from MRI. MC-DDPM produces high-quality sCT images and is reported as superior to several competing synthesis algorithms while eliminating the need for CT scans in radiation therapy planning.
- 7. Conclusion: The method is a 3D MRI-to-CT denoising diffusion probabilistic model for generating synthetic CT scans from MR scans.It uses a diffusion process to produce CT images from MRI inputs.
- 7. Conclusion: MC-DDPM uses a 3D shifted-window transformer network to learn the diffusion process that converts Gaussian noise into a realistic CT scan from a given MR scan.The model's reverse generation process is conditioned on the MRI input.
- 7. Conclusion: The generated sCT images show superior image quality compared with several competing state-of-the-art synthesis algorithms, including GANs and conventional DDPMs.The comparison is stated at the method level rather than for a single metric.
- 7. Conclusion: MC-DDPM generates high-quality sCT images using only MRI inputs, eliminating the need for CT scans in radiation therapy planning.The paper frames this as improving the quality of patient care.