Source-linked AI summary
Deep CT to MR Synthesis using Paired and Unpaired Data
Cheng-Bin Jin, Hakil Kim, Wonmo Jung, Seongsu Joo, Ensik Park, Ahn Young Saem, In Ho Han, Jae Il Lee, Xuenan Cui
TL;DR
The paper addresses limitations of CT-to-MR synthesis involving paired-data registration and blurred outputs, alongside challenges in using unpaired data. It introduces MR-GAN, which combines paired and unpaired training with multiple losses, and reports superior qualitative and quantitative results, including the lowest MAE and highest PSNR.
Problem
CT-to-MR synthesis is limited by rigid registration requirements in paired training, context misalignment in unpaired training, and potentially blurry generated outputs.
Method
MR-GAN combines paired and unpaired data with adversarial, dual cycle-consistent, and voxel-wise losses to synthesize MR images from CT images.
Results
The proposed system produced realistically-looking MR images and achieved the lowest MAE of 19.36 ± 2.73 and highest PSNR of 65.35 ±.
Takeaways & Limitations
Using paired and unpaired data together addresses the context-misalignment problem of unpaired training while alleviating paired-training registration and blurring issues.
Takeaways & Limitations
Paired training is limited by blurry output due to unpaired data in the experiments.
Abstract
from arXiv · showhide
MR imaging will play a very important role in radiotherapy treatment planning for segmentation of tumor volumes and organs. However, the use of MR-based radiotherapy is limited because of the high cost and the increased use of metal implants such as cardiac pacemakers and artificial joints in aging society. To improve the accuracy of CT-based radiotherapy planning, we propose a synthetic approach that translates a CT image into an MR image using paired and unpaired training data. In contrast to the current synthetic methods for medical images, which depend on sparse pairwise-aligned data or plentiful unpaired data, the proposed approach alleviates the rigid registration challenge of paired training and overcomes the context-misalignment problem of the unpaired training. A generative adversarial network was trained to transform 2D brain CT image slices into 2D brain MR image slices, combining adversarial loss, dual cycle-consistent loss, and voxel-wise loss. The experiments were analyzed using CT and MR images of 202 patients. Qualitative and quantitative comparisons against independent paired training and unpaired training methods demonstrate the superiority of our approach.
I. INTRODUCTION
The paper motivates CT-to-MR synthesis because MR-based radiotherapy is costly and constrained by metal implants, while CT is faster and higher-resolution. It combines paired and unpaired data to address registration, blurring, and context-misalignment problems.
- Motivation: CT offers faster imaging, higher resolution, and reduced motion artifact compared with MR imaging.CT can also differentiate soft tissue, especially with intravenous contrast.
- Contributions: The paper presents CT-to-MR synthesis as a new approach and states that the framework can extend to MR-CT and CT-PET synthesis.The introduction identifies this as the first study attempting CT-to-MR translation, to the authors’ knowledge.
- Proposed approach: The proposed MR-GAN trains on paired and unpaired data using adversarial, dual cycle-consistent, and voxel-wise losses.The approach is intended to alleviate paired-training registration and blurring while overcoming unpaired-training context misalignment.
- Existing limitations: Paired training requires difficult and expensive rigid alignment, while voxel-wise losses can produce blurry synthesized images.Voxel-wise losses also depend on large numbers of aligned CT and MR images.
- Existing limitations: Unpaired training provides abundant data and more realistic images, but cycle-consistent loss can misrepresent soft-tissue and blood-vessel context.Unpaired images may come from different patients and anatomical locations, creating context-misalignment concerns.
II. MATERIALS AND METHODS
The study uses brain CT and MR data from 202 patients, with most patients contributing unpaired data and 20 contributing paired scans. Images were processed and registered before synthesis experiments.
- Data acquisition: The dataset comprised brain CT and MR images from 202 patients scanned for brain-tumor radiotherapy planning.Among them, 98 patients had only CT images and 84 had only MR images.
- Data acquisition: The remaining 20 patients provided paired CT and MR images acquired during radiation treatment.Paired sets were formed by aligning and registering same-patient CT and MR images using affine transformation based on mutual information.
- Preprocessing: CT and MR images were resampled to 1.00 × 1.00 × 1.00 mm3 before registration.CT skull masking and MR skull stripping were performed before affine registration.
- Preprocessing: The CT images were windowed with a length of 80 Hounsfield units and a center of 40 HU to maximize information inside the brain.After registration, CT and MR images were reported to be well aligned spatially and temporally.
II.B. MR-GAN
MR-GAN extends the CycleGAN structure with dual paired and unpaired cycles. Paired cycles add pair-aware discrimination and voxel-wise supervision, while unpaired cycles retain cycle-consistency and adversarial training.
- MR-GAN architecture: MR-GAN uses four cycles: forward and backward cycles for both unpaired and paired training data.This dual cycle-consistent term is the framework’s central design choice.
- Unpaired-data cycles: In the forward unpaired cycle, SynMR translates CT into realistic MR images while DisMR distinguishes synthesized images from real MR images.Cycle-consistency translates the synthesized MR image back to CT and compares it with the original CT image.
- Unpaired-data cycles: The backward unpaired cycle translates MR images to CT images and operates in the opposite direction of the forward cycle.The backward cycle is also enforced to improve training stability.
- Paired-data cycles: Paired-data discriminators assess CT–MR pairs rather than only judging individual real and synthesized images.This pair-aware discrimination is combined with voxel-wise loss between synthesized and reference images.
- Paired-data cycles: The SynMR and SynCT networks operate in paired-data cycles as they do in the corresponding unpaired-data cycles.The paired cycles add paired supervision without changing the basic synthesis-network roles.
II.C. Objective
MR-GAN combines adversarial, dual cycle-consistent, and voxel-wise losses to translate between CT and MR while using both paired and unpaired data.
- II.C. Objective: MR-GAN trains SynMR and SynCT with adversarial losses against discriminators DisMR and DisCT.The synthesis networks translate between CT and MR domains, while the discriminators distinguish synthesized from real images.
- II.C. Objective: The dual cycle-consistent term includes forward and backward cycles for both paired and unpaired training data.The four cycles are forward unpaired-data, backward unpaired-data, forward paired-data, and backward paired-data cycles.
- II.C. Objective: Unpaired-data cycles reconstruct images across domains, while paired-data cycles additionally classify real and synthesized image pairs.For paired data, voxel-wise loss between synthesized and reference images is also included.
- II.C. Objective: Least-squares loss stabilizes unpaired-data training, whereas retaining negative log-likelihood for paired data produced higher-quality results.The paired-data objective was therefore kept unchanged in this respect.
- II.C. Objective: The voxel-wise L1 loss makes synthesized images close to paired reference images in both translation directions.The same constraint is applied to SynCT even though SynMR is the final synthesis product, supporting higher-quality synthesized MR images.
- II.C. Objective: The overall objective weights adversarial, dual cycle-consistent, and voxel-wise losses through λ and γ.The optimization alternates discriminator and synthesis-network updates for both MR and CT directions.
II.D. Implementation
The implementation uses convolutional synthesis networks and PatchGAN-style discriminators, with architecture and optimization choices adapted for paired and unpaired inputs.
- II.D. Implementation: SynMR and SynCT use a 2D fully convolutional architecture with strided convolutions, residual blocks, and upsampling layers.The networks accept 256 × 256 inputs and produce outputs of the same size.
- II.D. Implementation: DisMR and DisCT use PatchGANs that classify N × N image patches as real or fake.This patch-based design focuses the discriminators on high-frequency information in local image regions.
- II.D. Implementation: DisMR includes an extra convolutional head and tail to accommodate different input and loss functions for paired and unpaired data.DisCT uses the same discriminator architecture as DisMR.
- II.D. Implementation: At inference, only SynMR is run to translate CT slices into MR images.The reverse network SynCT is used during training but is not required as the final synthesis product.
III.A. Data preprocessing
The study uses paired and unpaired brain CT/MR data from 202 patients, with separate paired training and testing subsets and standardized preprocessing.
- III.A. Data preprocessing: All unpaired data from the 202-patient dataset were used for training, while paired data were split across training and testing patients.Paired training used 10 patients, and a separate test set contained corresponding CT and reference MR images from 10 patients.
- III.A. Data preprocessing: Volumes were resampled to 256 × 256 pixels with 256 grayscale levels and uniformly distributed intensity values.The preprocessing was applied to both HU-based CT and MR data.
- III.A. Data preprocessing: Unpaired CT and MR images were augmented independently, whereas paired images received the same augmentation factor.Training augmentation included horizontal flipping and random cropping from padded images.
- III.A. Data preprocessing: Training took about 72 hours for 3e5 iterations on a single GeForce GTX 1080Ti GPU.At inference, one CT slice required 35 ms for MR synthesis.
III.B. Evaluation metrics
Synthesis quality is evaluated by comparing synthesized and reference MR images using mean absolute error and peak signal-to-noise ratio.
- III.B. Evaluation metrics: Mean absolute error (MAE) measures the average pixel distance between synthesized and reference MR images.The computation averages absolute differences over aligned voxels and image slices.
- III.B. Evaluation metrics: Peak signal-to-noise ratio (PSNR) measures the ratio between maximum possible intensity and the synthesized-reference mean square error.The maximum intensity is set to MAX = 255.
- III.B. Evaluation metrics: Figure 5 presents input CT, synthesized MR, reference MR, and their absolute error images from left to right.The visual comparison focuses on correspondence between synthesized and reference MR images and the associated error pattern.
III.C. Analysis of MR synthesis using paired and unpaired data
The proposed method was compared with independently trained paired and unpaired systems using quantitative and qualitative evaluations. Joint paired-and-unpaired training achieved the strongest reported performance and produced images closer to reference MR images.
- MR-GAN differentiated anatomical structures with similar CT intensity; differences were largest in bony structures and smallest in soft brain tissues.The compared structures included bones, gyri, and soft brain tissue.
- The method combines adversarial, dual cycle-consistent, and paired-data voxel-wise losses, while using shared architecture and implementation details for baseline comparisons.The paired baseline used adversarial and voxel-wise losses, whereas the unpaired baseline used CycleGAN cycle consistency.
- Limited-data paired training outperformed unpaired CycleGAN training, although paired outputs were blurry and unpaired outputs lost soft-tissue anatomy and included bony-structure artifacts.These qualitative differences motivated combining paired and unpaired data.
- The proposed method had the best performance across measurements, with the lowest MAE and highest PSNR among the compared training strategies.The comparison included independent paired training, independent unpaired training, and joint paired-and-unpaired training.
- Joint paired-and-unpaired training produced results that closely approximated reference MR images and were clearer for some details.The qualitative comparison covered input CT, synthesized MR images from each method, and reference MR images.
- MR-GAN reconstructed CT images close to the inputs, with differences concentrated around bone contours and smoother reconstructions than CycleGAN.The analysis used reconstructed CT images and relative difference maps after CT-to-MR-to-CT translation.
IV. DISCUSSION
The discussion presents a system that synthesizes MR images from CT using paired and unpaired data. It reports stronger quantitative and qualitative correspondence than comparison methods, while noting that numerical metrics may not fully reflect perceived image quality.
- The method combines adversarial loss, dual cycle-consistent loss, and paired-data voxel-wise loss to generate realistic-looking MR images.
- 19.36 ± 2.73 MAE and 65.35 ± 0.86 PSNR were achieved, respectively, as the lowest MAE and highest PSNR in the reported comparison.The quantitative evaluation found synthesized images closer to reference MR images than those from other methods.
- The authors caution that numerical differences in quantitative evaluation do not always indicate qualitative differences correctly and propose future perceptual studies.
- The system may support CT-based radiotherapy planning for patients unable to undergo MR scanning because of pacemakers, metal implants, or limited medical services.
- The system can synthesize MR images from paired data, unpaired data, or both, with joint training producing higher-quality images than either data type alone.The approach is intended to address limitations associated with rigidly aligned paired data and context-misaligned unpaired data.
V. CONCLUSION
The proposed CT-to-MR synthesis system combines paired and unpaired data to address limitations of each training setting. Tests indicate that MR-GAN produced images closer to reference MR images and translated complex structures in 2D brain slices.
- The system synthesizes MR images from CT images using paired and unpaired data together.
- It addresses context misalignment in unpaired training while alleviating rigid registration and blurred results associated with paired training.
- Unpaired data, combined with limited paired data, could support effective synthesis in many cases.
- MR-GAN outputs were closer to reference MR images than outputs from other methods on the test set.
- The system efficiently translated structures including soft brain vessels, gyri, and bones in complicated 2D brain slices.
- Future work will investigate three-dimensional anatomical information represented in CT and MR images.