PaperlayerView exact revision on arXiv

Source-linked AI summary

Image Synthesis in Multi-Contrast MRI with Conditional Generative Adversarial Networks

Salman Ul Hassan Dar, Mahmut Yurt, Levent Karacan, Aykut Erdem, Erkut Erdem, Tolga Çukur

arXiv:1802.01221v1cs.CV

TL;DR

Limited scan time can prevent acquisition of all useful MRI contrasts, while existing synthesis methods may lose high-spatial-frequency information. This paper uses conditional GANs with adversarial, pixel-wise, and cycle-consistency losses for registered and unregistered synthesis, and reports higher synthesis quality than prior methods across three brain MRI datasets. The approach is intended to improve the quality and versatility of multi-contrast MRI exams without prolonged examinations.

  • Problem

    Scan-time limitations can prevent acquisition of all diagnostic MRI contrasts, and existing nonlinear or deterministic synthesis methods may lose high-spatial-frequency information.

  • Method

    Conditional GANs synthesize target contrasts from source contrasts, using adversarial loss with pixel-wise loss for registered images, cycle-consistency loss for unregistered images, and neighboring cross-sections.

  • Results

    The proposed methods achieved higher synthesis quality than state-of-the-art methods across registered and unregistered scenarios evaluated on three publicly available brain MRI datasets.

  • Takeaways & Limitations

    The approach can improve the quality and versatility of multi-contrast MRI exams without requiring prolonged examinations.

  • Takeaways & Limitations

    The performance of conditional GAN architectures with missing source contrasts requires further investigation.

Abstract

from arXiv · show

Acquiring images of the same anatomy with multiple different contrasts increases the diversity of diagnostic information available in an MR exam. Yet, scan time limitations may prohibit acquisition of certain contrasts, and images for some contrast may be corrupted by noise and artifacts. In such cases, the ability to synthesize unacquired or corrupted contrasts from remaining contrasts can improve diagnostic utility. For multi-contrast synthesis, current methods learn a nonlinear intensity transformation between the source and target images, either via nonlinear regression or deterministic neural networks. These methods can in turn suffer from loss of high-spatial-frequency information in synthesized images. Here we propose a new approach for multi-contrast MRI synthesis based on conditional generative adversarial networks. The proposed approach preserves high-frequency details via an adversarial loss; and it offers enhanced synthesis performance via a pixel-wise loss for registered multi-contrast images and a cycle-consistency loss for unregistered images. Information from neighboring cross-sections are utilized to further improved synthesis quality. Demonstrations on T1- and T2-weighted images from healthy subjects and patients clearly indicate the superior performance of the proposed approach compared to previous state-of-the-art methods. Our synthesis approach can help improve quality and versatility of multi-contrast MRI exams without the need for prolonged examinations.

1 - Introduction

Multi-contrast MRI provides complementary diagnostic information, but scan-time constraints and registration assumptions limit acquisition and existing synthesis methods. The paper proposes conditional GANs with scenario-specific losses and neighboring-slice information to improve target-contrast synthesis.

  • Motivation: Multi-contrast MRI captures complementary tissue information, increasing diagnostic information from the same anatomy.T1-weighted images delineate gray and white matter, whereas T2-weighted images delineate fluid from cortical tissue.
  • Motivation: Prolonged examinations and uncooperative patients can make acquiring a full set of contrasts impractical.Shorter-scan contrasts may therefore be preferred.
  • Prior limitations: Existing neural-network synthesis can perform well with missing source contrasts but assumes spatially registered multi-contrast images and may miss high-frequency errors.Multimodal uses mean squared or absolute error losses, which can perform poorly at high spatial frequencies.
  • Proposed approach: The proposed conditional GANs synthesize target contrasts from source contrasts, using pixel-wise loss for registered images and cycle-consistency loss for unregistered images.Neighboring cross-sections are also incorporated during synthesis.
  • Contributions: The study reports conditional GAN synthesis using adversarial, pixel-wise, and cycle-consistency losses, with neighboring cross-sections providing additional information.The listed contributions include end-to-end synthesis and unregistered-image synthesis.

2 - Methods

The method applies conditional adversarial networks to translate source-contrast MR images into target contrasts. It uses adversarial loss for realistic high-frequency detail, pixel-wise loss for registered data, and cycle consistency when images are unregistered.

  • Adversarial loss: GAN training jointly optimizes a generator to produce realistic images and a discriminator to distinguish generated images from real images.The adversarial loss supports modeling high-spatial-frequency information.
  • Conditional adversarial synthesis: Conditional GANs translate an input source image into a target-domain image while conditioning the synthesis on the source image.Image-to-image translation transforms different representations of the same underlying scene or anatomy.
  • Registered images: For registered source and target images, pGAN uses pixel-wise L1 loss between synthesized and true target images.The complete loss combines adversarial and pixel-wise terms, weighted by a relative-loss parameter.
  • Unregistered images: For unregistered images, cGAN replaces pixel-wise supervision with cycle consistency because corresponding pixels are not aligned.The cycle loss constrains synthesis by requiring the source image to be reconstructable from the synthesized target.
  • Unregistered images: The unregistered cGAN uses two generators and two discriminators to model mappings between source and target contrasts.The target-domain generator produces images resembling the target contrast, while the corresponding discriminator distinguishes them from real target images.

2.2 – MRI datasets

Experiments used registered and unregistered multi-contrast brain MRI from three datasets covering healthy subjects and patients with structural lesions. Dataset sizes, splits, and acquisition protocols varied substantially.

  • Datasets: Experiments evaluated pGAN and cGAN across the MIDAS, IXI, and BRATS brain MRI datasets.Registered data were used for both pGAN and cGAN, whereas unregistered data were used only for cGAN.
  • Datasets: MIDAS and IXI primarily contain healthy-subject data, while BRATS contains data from patients with structural lesions.The datasets therefore cover both healthy and pathological brain MRI.
  • MIDAS: MIDAS contributed 66 subjects and 4,865 T1-weighted and 4,865 T2-weighted images, with 3,774 images per contrast for training and 1,091 for testing.These images were acquired using specified T1- and T2-weighted protocols.
  • IXI: IXI contributed 30 subjects, with 3,320 T1-weighted images and 2,730 T2-weighted images divided into training and testing sets.The splits were 2,780/540 for T1-weighted images and 2,275/455 for T2-weighted images.
  • BRATS: BRATS contributed 28 subjects with visible lesions and 2,828 T1-weighted and 2,828 T2-weighted images, split into 2,424 training and 404 testing images per contrast.BRATS data came from different scanning protocols at separate sites, so no common acquisition protocol existed.

2.3 – Image registration

The registered-image experiments required preprocessing because MIDAS and IXI source and target images were originally unregistered. Dataset-specific mutual-information registration methods were selected before network training.

  • Registration setup: Multi-contrast images were assumed to be registered in the registered-image synthesis scenario.This assumption defined the setup for training and evaluating registered-image models.
  • Dataset preprocessing: MIDAS and IXI images were registered before network training because their T1- and T2-weighted images were unregistered.MIDAS used affine registration, while IXI used rigid registration.
  • Registration methods: Both datasets used mutual-information costs to select the registration transformation.Affine transformation performed better for MIDAS, whereas rigid transformation performed better for IXI.

2.4 – Network training

The paper trains pGAN for aligned source–target images and cGAN for unregistered images, with single- and multi-cross-section variants. Training uses a standard adversarial-network implementation with fixed and decayed learning rates.

  • pGAN learns source-to-target contrast mappings assuming perfect alignment between the images.
  • Both architectures use single-cross-section and three-consecutive-cross-section input variants.
  • Training uses 256x256 zero-padded images, 200 epochs, Adam optimization, and minibatches of one.
  • The learning rate is fixed at 0.0002 for the first 100 epochs and linearly decayed to zero over the remaining 100.
  • cGAN learns mappings between unregistered source and target images without assuming spatial alignment.

2.5 – Competing methods

The experiments compare the proposed models with Replica and Multimodal, using matched training and test data and two-contrast inputs for fair comparison.

  • Replica estimates nonlinear source-to-target mappings from multiscale image patches using multiresolution random forests.
  • Multimodal uses an end-to-end neural network to predict a target contrast from source images.
  • All compared methods use the same training and test data, while Replica and Multimodal are restricted to two contrasts.

2.6 - Experiments

The experiments test registration direction, synthesis direction, cross-section context, and performance against competing methods across registered and unregistered datasets.

  • The study evaluates whether registration direction affects synthesis quality using T1- and T2-weighted registered datasets.
  • Unregistered datasets produce four synthesis cases combining T1/T2 source contrasts, registered targets, and both synthesis directions.
  • Models receive either one cross-section or three consecutive cross-sections, yielding 8 pGAN and 4 cGAN models for the unregistered cases.
  • The proposed pGAN and cGAN models are compared with Replica and Multimodal on matched data.
  • Synthesis quality is assessed with PSNR and SSIM between normalized synthesized images and reference targets.

3 – Results

Across healthy and pathological MRI datasets, pGAN generally provides the strongest synthesis results, while multi-cross-section input improves pGAN performance in noisy settings.

  • MIDAS: 2.72 dB and 0.55 dB: pGAN exceeds cGANreg in average PSNR for T2 and T1 synthesis, respectively, on MIDAS.
  • Cross-section context: 0.55 dB and 0.99 dB: multi-cross-section pGAN improves PSNR over single-cross-section pGAN for T2 and T1 synthesis, respectively.
  • IXI: 1.04 dB and 2.41 dB: pGAN exceeds Multimodal in average PSNR for T2 and T1 synthesis, while outperforming competing methods across examined IXI cases.
  • BRATS: 2.34 dB and 2.27 dB: pGAN exceeds Multimodal in average PSNR for T2 and T1 synthesis on pathological BRATS images.
  • BRATS: 1.65 dB and 2.53 dB: pGAN exceeds Replica in average PSNR for T2 and T1 synthesis on BRATS images.

4 – Discussion

The discussion positions conditional GANs as an end-to-end alternative that captures high-frequency detail, supports registered and unregistered training, and can be extended to broader inputs and modalities.

  • Discussion: The proposed methods achieved higher visual and quantitative synthesis quality than state-of-the-art methods across registered and unregistered training scenarios.Evaluation used three publicly available brain MRI datasets and PSNR and SSIM metrics.
  • Discussion: Registered synthesis combines adversarial and pixel-wise losses, whereas unregistered synthesis uses cycle-consistency loss to learn from unregistered images.The approach also uses correlated structure across neighboring cross-sections.
  • Discussion: Conditional GANs learn the target contrast distribution from source images, using adversarial loss to capture detailed texture information and improve synthesis quality.This contrasts with typical squared or absolute error losses used in earlier neural-network methods.
  • Discussion: When source and target voxel sizes differ substantially, multimodal registration followed by a cascaded spatial transformation may alleviate interpolation-related performance loss.The authors note that choosing the optimal registration direction remains challenging because reference images differ between directions.
  • Discussion: Both pGAN and cGAN can be extended to multiple source contrasts and cross-sections, potentially improving synthesis when some source contrasts are unavailable.The performance of conditional GANs with missing inputs remains an open area for investigation.
  • Discussion: cGAN permits training with unregistered and unpaired multi-contrast datasets, which may facilitate compiling larger datasets for deeper networks.The study itself used paired images for unbiased comparison, while mixed paired and unpaired training is suggested for future improvement.

5 – Conclusion

The paper concludes that conditional GANs provide a multi-contrast MRI synthesis approach that improves high-frequency accuracy and overall quality across healthy-subject and glioma-patient datasets.

  • 5 – Conclusion: The method performs end-to-end conditional GAN training to synthesize a target contrast from source-contrast images.It combines adversarial loss, pixel-wise loss for registered images, cycle-consistency loss for unregistered images, and neighboring cross-section information.
  • 5 – Conclusion: Adversarial loss improves synthesis accuracy for high-spatial-frequency information in the target contrast.Pixel-wise and cycle-consistency losses provide additional improvements for registered and unregistered settings, respectively.
  • 5 – Conclusion: The proposed method outperformed two state-of-the-art synthesis methods on multi-contrast brain MRI datasets from healthy subjects and glioma patients.The conclusion states that the approach holds promise for multi-contrast synthesis in clinical practice.

TABLES

The tables evaluate conditional GAN variants and prior methods for synthesizing T1- and T2-weighted images across three brain MRI datasets and multiple registration settings.

  • Table I: Table I reports MIDAS synthesis quality for cGAN trained on unregistered images, cGAN trained on registered images, and pGAN.It covers T1- and T2-weighted synthesis in both registration directions and reports SSIM and PSNR.
  • Table II: Table II reports MIDAS synthesis using three consecutive cross-sections and summarizes SSIM and PSNR as mean±std across test images.The table compares cGANunreg, cGANreg, and pGAN across T1- and T2-weighted tasks.
  • Table III: Table III compares cGANreg, pGAN, Replica, and Multimodal on T1- and T2-weighted synthesis in the IXI dataset.Results are reported with SSIM and PSNR for two registration directions per synthesis task.
  • Table IV: Table IV compares cGANreg, pGAN, Replica, and Multimodal on T1- and T2-weighted synthesis in the BRATS dataset.SSIM and PSNR are reported for two registration directions per synthesis task, with highest measurements marked in bold.
Loading 1802.01221v1…