Source-linked AI summary
Deep MR to CT Synthesis using Unpaired Data
Jelmer M. Wolterink, Anna M. Dinkla, Mark H. F. Savenije, Peter R. Seevinck, Cornelis A. T. van den Berg, Ivana Isgum
TL;DR
MR-only radiotherapy needs accurate synthetic CT while current deep-learning approaches rely on paired, aligned MR–CT data. The paper trains a bidirectional CycleGAN with unpaired brain MR and CT images, and reports close agreement with reference CT images plus better results than paired training. The study also identifies limitations related to same-patient data and the use of 2D rather than 3D information.
Problem
Existing MR-to-CT synthesis methods require paired MR and CT volumes, while misalignment and scarce paired data constrain this training setup.
Method
A bidirectional CycleGAN with two synthesis CNNs, two discriminator CNNs, adversarial feedback, and cycle consistency translates 2D brain MR slices into CT slices and vice versa.
Results
73.7 ± 2.3 HU MAE and 32.3 ± 0.7 PSNR were obtained on an independent test set of six images, while unpaired training significantly outperformed the paired model.
Takeaways & Limitations
Unpaired and unaligned training can synthesize realistically looking brain CT images and may support MR-only radiotherapy and other single-modality clinical settings.
Takeaways & Limitations
The study used MR and CT images from the same patients and 2D synthesis; future work will assess 3D information and dose-planning accuracy.
Abstract
from arXiv · showhide
MR-only radiotherapy treatment planning requires accurate MR-to-CT synthesis. Current deep learning methods for MR-to-CT synthesis depend on pairwise aligned MR and CT training images of the same patient. However, misalignment between paired images could lead to errors in synthesized CT images. To overcome this, we propose to train a generative adversarial network (GAN) with unpaired MR and CT images. A GAN consisting of two synthesis convolutional neural networks (CNNs) and two discriminator CNNs was trained with cycle consistency to transform 2D brain MR image slices into 2D brain CT image slices and vice versa. Brain MR and CT images of 24 patients were analyzed. A quantitative evaluation showed that the model was able to synthesize CT images that closely approximate reference CT images, and was able to outperform a GAN model trained with paired MR and CT images.
1 Introduction
MR-only radiotherapy could avoid separate CT acquisition, but existing synthesis methods rely on paired, aligned MR–CT data vulnerable to misalignment and limited availability. This study uses CycleGAN with unpaired data to synthesize brain CT from MR and reports improved agreement relative to paired training.
- Motivation: MR-only treatment planning seeks to derive a synthetic CT from MR while avoiding separate CT acquisition.Separate MR and CT acquisition is time-consuming, costly, and burdensome, and MR–CT alignment may be compromised.
- Limitations of paired training: Voxel-wise CNN training uses rigidly aligned reference CT volumes, but residual misalignment can produce blurred synthesized images.Local misalignment may remain in regions including the throat, mouth, vertebrae, and nasal cavities despite good skull alignment.
- Limitations of paired training: Conventional adversarial synthesis reduces reliance on purely voxel-wise appearance but still requires paired MR and CT volumes.Additional unpaired scans could expand training data when patients were scanned for different purposes or lack both modalities.
- Proposed direction: Unpaired image-translation methods use adversarial image-quality feedback and cycle consistency to preserve similarity to the input image.Cycle consistency discourages outputs that look realistic but bear little similarity to the source image.
- Contribution: The study applies CycleGAN to brain MR-to-CT synthesis and shows that spatially aligned same-patient pairs are not necessary for training.Figure 1 contrasts paired slices from the same patient and location with unpaired slices from different patients and brain locations.
2 Data
The study analyzed brain MR and CT images from 24 radiotherapy patients scanned on the same day in treatment position. Patients with heavy dental artifacts were excluded.
- Cohort: The dataset comprised brain MR and CT images from 24 patients scanned for brain-tumor radiotherapy planning.MR and CT were acquired on the same day in radiation treatment position using a thermoplastic immobilization mask.
- Cohort: Patients with heavy dental artifacts on CT and/or MR were excluded.
- Image acquisition: MR images were T1-weighted 3D acquisitions from a Philips Ingenia 1.5T scanner, while CT images were acquired helically on a Philips Brilliance Big Bore scanner.
3 Methods
The method uses bidirectional CycleGAN training with synthesis and discriminator CNNs, adversarial objectives, and cycle-consistency losses. Experiments synthesize and evaluate 2D brain CT slices against aligned reference CT images.
- CycleGAN architecture: CycleGAN uses forward and backward cycles to translate between MR and CT domains.The same SynCT and SynMR networks are used for both directions, with DisCT and DisMR judging synthesized images in their respective domains.
- Forward cycle: In the forward cycle, SynCT maps MR to CT, SynMR reconstructs MR, and DisCT distinguishes synthesized from real CT images.The reconstruction objective encourages the original MR image to be recovered after the two translations.
- Backward cycle: In the backward cycle, SynMR maps CT to MR and SynCT reconstructs CT while DisMR distinguishes synthesized from real MR images.The backward cycle is trained to improve training stability.
- Adversarial objectives: The discriminator minimizes classification loss, whereas the synthesis network maximizes it by producing images indistinguishable from real images.DisCT predicts 1 for real CT and 0 for synthesized CT; the corresponding MR discriminator uses the analogous labels.
- Cycle consistency: Additional losses enforce bidirectional cycle consistency by penalizing differences between original and reconstructed images.A weighting parameter λ is added to the SynCT and SynMR loss functions.
- Implementation: The implementation performs voxel regression and image classification on 2D sagittal slices using fully convolutional synthesis and discriminator networks.SynCT and SynMR use two strided convolution layers, nine residual blocks, and two fractionally strided convolution layers; inputs and outputs are 256 × 256 pixels.
- Evaluation: Evaluation compares synthetic and real CT images using mean absolute error and peak-signal-to-noise ratio within a head-region mask.The voxel-wise comparison uses aligned voxels and excludes surrounding air.
4 Experiments and Results
The experiments evaluate unpaired MR-to-CT synthesis on held-out patient data and compare it with paired training. Results include quantitative agreement, visual quality, and bidirectional reconstruction consistency.
- Data and evaluation: The dataset was split into 18 training patients and 6 independent test patients.
- Data and evaluation: Each volume contained 183 sagittal 2D slices resampled to 256 × 256 pixels, with random cropping used for augmentation.
- Qualitative results: The synthesized CT differentiated structures with similar MR intensities, while errors were greatest in bone structures and partly reflected image quality and misalignment.
- Comparison with paired training: Unpaired training produced significantly better agreement than paired training, whose images were blurrier and contained a high-intensity neck artifact.
- Cycle-consistency evaluation: Cycle consistency yielded reconstructed MR images that were remarkably similar to the originals, with small, homogeneously distributed differences.
5 Discussion and Conclusion
The study demonstrates CT synthesis from unpaired and unaligned MR and CT images using adversarial discrimination and cycle consistency. The model achieved quantitative correspondence with reference CT images and outperformed paired-data training in the study’s test sets, while remaining subject to evaluation and training-scope limitations.
- 5 Discussion and Conclusion: Unpaired and unaligned MR and CT images were sufficient to train a CNN for realistic CT synthesis.The model used adversarial discriminator feedback and similarity of back-transformed outputs to the original MR inputs.
- 5 Discussion and Conclusion: 73.7 ± 2.3 HU (MAE) and 32.3 ± 0.7 (PSNR) measured average correspondence between synthetic and reference CT images on an independent six-image test set.Direct comparison with prior studies was infeasible because they used different datasets and anatomical coverage.
- 5 Discussion and Conclusion: The unpaired-data model outperformed the paired-data model, producing CT images with fewer artifacts and less blurring in qualitative analysis.The authors suggest this could be related to MR–CT misalignment being ignored during unpaired training.
- 5 Discussion and Conclusion: Unaligned-data training may support MR-only radiotherapy planning and synthesis between modalities when paired acquisitions are scarce.Proposed applications include synthesis across MR field strengths and CT dose levels.
- 5 Discussion and Conclusion: The study successfully synthesized 2D medical images, but whether 3D information improves performance remains future work.The current results nevertheless translated structures with complex 3D appearance, including vertebrae and bones.
- 5 Discussion and Conclusion: Training used MR and CT images from the same patients, so extending the method to disjoint patient sets remains unresolved.The authors note that corresponding reference images could occasionally provide stronger discriminator feedback.