Source-linked AI summary
Medical Image Synthesis with Context-Aware Generative Adversarial Networks
Dong Nie, Roger Trullo, Caroline Petitjean, Su Ruan, Dinggang Shen
TL;DR
The paper addresses estimating clinically useful CT images from safer MRI despite the difficult cross-modality mapping. It trains a context-aware adversarial 3D FCN with gradient-difference and reconstruction objectives, and reports superior performance against three compared methods on two datasets.
Problem
Estimating CT from MRI is clinically motivated because CT supports dose calculation and PET attenuation correction but exposes patients to radiation, while MRI cannot directly provide those functions.
Method
A 3D fully convolutional GAN generates CT from MR patches using adversarial and reconstruction losses, image gradient-difference constraints, and Auto-Context for enlarged context.
Results
The method outperforms three compared state-of-the-art methods on two real CT/MR datasets.
Takeaways & Limitations
The proposed model is presented as suitable for CT prediction and related generative medical-image tasks such as super-resolution and denoising.
Abstract
from arXiv · showhide
Computed tomography (CT) is critical for various clinical applications, e.g., radiotherapy treatment planning and also PET attenuation correction. However, CT exposes radiation during acquisition, which may cause side effects to patients. Compared to CT, magnetic resonance imaging (MRI) is much safer and does not involve any radiations. Therefore, recently, researchers are greatly motivated to estimate CT image from its corresponding MR image of the same subject for the case of radiotherapy planning. In this paper, we propose a data-driven approach to address this challenging problem. Specifically, we train a fully convolutional network to generate CT given an MR image. To better model the nonlinear relationship from MRI to CT and to produce more realistic images, we propose to use the adversarial training strategy and an image gradient difference loss function. We further apply AutoContext Model to implement a context-aware generative adversarial network. Experimental results show that our method is accurate and robust for predicting CT images from MRI images, and also outperforms three state-of-the-art methods under comparison.
1 Introduction
CT is clinically important but exposes patients to radiation, while MRI is safer yet cannot provide CT’s treatment-planning and attenuation-correction information. The paper addresses the difficult MRI-to-CT mapping with a context-aware adversarial 3D FCN and reports superior prediction performance.
- Clinical motivation: CT supplies information needed for radiotherapy dose calculation and PET attenuation correction, but its radiation exposure motivates estimating CT from safer MRI.MRI also provides more anatomical detail for diagnosis, but cannot directly replace CT for these applications.
- Technical challenge: MRI-to-CT estimation is technically difficult because the modalities have very different appearances and MRI contains richer texture information.These differences make direct mapping challenging.
- Prior approaches: Existing approaches include tissue-segmentation methods that assign CT-related values to segmented MR tissue classes.The cited examples use fuzzy clustering or multi-class tissue segmentation before assignment.
- Methodological motivation: Conventional L2 losses can produce blurry outputs for multimodal data, motivating adversarial methods that generate more realistic images.The paper uses this motivation to move beyond a purely reconstruction-based objective.
- Proposed approach: The proposed method uses a 3D fully convolutional network as a GAN generator, combines adversarial and reconstruction objectives, and applies Auto-Context to enlarge context.The FCN produces structured outputs that preserve neighborhood information, while Auto-Context makes the patch-based GAN context-aware.
- Evaluation: Experiments on two real CT/MR datasets show effective CT prediction and performance exceeding three compared state-of-the-art methods.The paper reports this as the overall experimental outcome.
2 Methods
The method uses a 3D fully convolutional generator within a GAN to estimate CT from MRI while preserving volumetric neighborhood information. It combines adversarial, reconstruction, and gradient-difference objectives with AutoContext refinement for patch-based prediction.
- Generative adversarial network: A 3D fully convolutional network generates structured CT outputs from MRI while modeling spatial information across slices.Unlike conventional CNNs, the FCN produces structured outputs that preserve neighborhood information.
- Generative adversarial network: The GAN trains a generator to produce realistic CT and a discriminator to distinguish real CT from generated CT.The discriminator uses binary cross entropy, while the generator combines adversarial and reconstruction terms.
- Loss functions: Gradient-difference loss minimizes discrepancies between generated and ground-truth gradient magnitudes, helping preserve strong edges alongside the L2 term.The loss is approximated with finite differences and included in the generator’s weighted total loss.
- Training: Training alternates discriminator updates on real and generated CT minibatches with generator updates using paired MRI and CT samples.This alternating procedure separately updates D and G with their respective minibatches.
- Network architecture: The generator avoids pooling and uses batch normalization to retain spatial resolution and support stable adversarial training.The architecture contains convolution, batch-normalization, and ReLU stages, followed by a CT-estimation output layer.
- AutoContext refinement: Because patch-based training limits context, AutoContext iteratively feeds previously estimated CT patches as a second MRI channel to subsequent GANs.Features extracted from the previously estimated CT can encode information unavailable from individual patch features.
3 Datasets
The study evaluates its method on separate brain and pelvic CT/MR datasets, with aligned and preprocessed images used for pelvic training.
- Two real CT/MR datasets are used to test the proposed methods.
- Brain dataset: The brain dataset contains paired MRI and CT scans from 16 subjects in the ADNI database.
- Pelvic dataset: The pelvic dataset contains paired MR and CT images from 22 subjects.
- Pelvic dataset: Pelvic CT images are manually aligned to MR images, then the paired volumes are cropped to 153 × 193 × 50 with 1 × 1 × 1 mm3 spacing.
4 Experimental Results
Experiments compare the proposed adversarial FCN with established methods and assess adversarial training, AutoContext refinement, visual quality, and quantitative accuracy across brain and pelvic datasets.
- Experimental setup: Training uses paired MRI and CT patches with Adam, a 10^-6 learning rate, β1 = 0.5, and mini-batches of 10.
- Experimental setup: The method is compared with atlas-based, sparse-representation, and structured-random-forest AutoContext methods using leave-one-out experiments.
- Evaluation metrics: MAE and PSNR are used to compare prediction performance across methods.
- GAN contribution: PSNR increases from 24.7 with traditional FCN to 25.9 with the proposed GAN without ACM, while GAN outputs show fewer artifacts.
- AutoContext refinement: Both MAE and PSNR improve gradually and consistently with ACM iterations, especially during the first two iterations, so two iterations are selected.
- Qualitative results: Visual comparisons show that the proposed method better preserves continuity, cohesion, and smoothness and produces CT images closer to the real CT than competing methods.
- Brain dataset: The proposed method outperforms all other methods on both PSNR and MAE in the brain-dataset quantitative comparison.
- Pelvic dataset: 34.1 average PSNR is achieved on the pelvic dataset versus 32.1 for SRF+, with the proposed method outperforming alternatives on MAE and PSNR.
5 Conclusions
The paper develops a context-aware 3D GAN for estimating CT images from MRI patches and reports strong performance across brain and pelvic datasets. AutoContext enlarges the effective training context, while the proposed method surpasses the compared methods quantitatively and visually.
- Method: The proposed 3D GAN takes MR image patches as input and produces corresponding CT patches.The model uses AutoContext to enlarge the GAN's context during training and make it context-aware.
- Quantitative evaluation: The proposed method outperforms the other methods in quantitative comparisons using PSNR and MAE.The brain-dataset comparison includes Atlas, SR, SRF+, and Proposed across 16 subjects.
- Visual evaluation: On a typical pelvic subject, the predicted CT images from the proposed method are visually consistent with the ground-truth CT.The figure compares original MR images, estimated CT images from multiple methods, and ground-truth CT images.
- Pelvic dataset: 34.1 PSNR Mean(std.) is achieved by the Proposed method on the pelvic dataset, exceeding SRF+ at 32.1.The corresponding median PSNR is 34.1 for Proposed versus 31.8 for SRF+.