Source-linked AI summary
mustGAN: Multi-Stream Generative Adversarial Networks for MR Image Synthesis
Mahmut Yurt, Salman Ul Hassan Dar, Aykut Erdem, Erkut Erdem, Tolga Çukur
TL;DR
Incomplete or corrupted multi-contrast MRI acquisitions motivate synthesis methods that recover missing contrasts from available source images. mustGAN combines multiple source-specific streams with a joint multi-source stream and adaptively fuses their features. Across reported MRI synthesis tasks, it produced higher-quality results than isolated one-to-one and many-to-one methods, while its current implementation assumes paired source-target images.
Problem
Scan time and patient motion can limit the number and quality of acquired MRI contrasts, motivating synthesis of missing or corrupted contrasts.
Method
mustGAN combines multiple one-to-one streams, a joint many-to-one stream, and an adaptively positioned fusion block to synthesize a target contrast.
Results
mustGAN achieved higher-quality multi-contrast MR synthesis than isolated one-to-one and many-to-one methods, with reduced noise and improved sharpness.
Takeaways & Limitations
mustGAN combines source-specific and shared information for synthesis across T1-, T2-, PD-weighted, and FLAIR MRI images.
Takeaways & Limitations
The current implementation assumes paired source-target images from the same subjects, while paired datasets may be limited for less common contrasts.
Abstract
from arXiv · showhide
Multi-contrast MRI protocols increase the level of morphological information available for diagnosis. Yet, the number and quality of contrasts is limited in practice by various factors including scan time and patient motion. Synthesis of missing or corrupted contrasts can alleviate this limitation to improve clinical utility. Common approaches for multi-contrast MRI involve either one-to-one and many-to-one synthesis methods. One-to-one methods take as input a single source contrast, and they learn a latent representation sensitive to unique features of the source. Meanwhile, many-to-one methods receive multiple distinct sources, and they learn a shared latent representation more sensitive to common features across sources. For enhanced image synthesis, here we propose a multi-stream approach that aggregates information across multiple source images via a mixture of multiple one-to-one streams and a joint many-to-one stream. The shared feature maps generated in the many-to-one stream and the complementary feature maps generated in the one-to-one streams are combined with a fusion block. The location of the fusion block is adaptively modified to maximize task-specific performance. Qualitative and quantitative assessments on T1-, T2-, PD-weighted and FLAIR images clearly demonstrate the superior performance of the proposed method compared to previous state-of-the-art one-to-one and many-to-one methods.
I. INTRODUCTION
Multi-contrast MRI provides complementary tissue information, but scan-time limits and motion artifacts can make acquiring complete, high-quality contrast sets impractical. mustGAN addresses this setting by combining one-to-one and many-to-one synthesis streams.
- Different MRI contrasts improve tissue differentiation and diagnostic information, including gray-white matter delineation in T1-weighted images and fluid-cortical tissue delineation in PD-weighted images.
- Scan-time limitations and patient-motion artifacts can make multi-contrast acquisitions impractical.
- Missing or corrupted contrast synthesis is presented as a tool for improving the clinical feasibility and utility of multi-contrast MRI.
- Prior methods use either one-to-one synthesis from a single source contrast or many-to-one synthesis from multiple source contrasts.
- mustGAN combines multiple one-to-one streams with a joint many-to-one stream, fusing complementary and shared features before target-image generation.
B. Conditional Generative Adversarial Networks
Conditional GANs use source images as prior information for image-to-image translation, with adversarial objectives and optional pixel-wise losses for spatially registered images.
- Conditional GANs map statistically dependent source and target images by providing the source image as prior information.
- For spatially registered source and target images, a pixel-wise loss can be added between generated and ground-truth images.
- The conditional GAN joint loss combines adversarial and pixel-wise objectives.
- With K source contrasts, a many-to-one conditional GAN receives the source images collectively, corresponding to input-level concatenation in a single-stream network.
III. METHODS
mustGAN is a multi-stream GAN that learns separate source-specific and joint multi-source representations, then adaptively fuses them and trains a joint network to synthesize the target contrast.
- mustGAN adaptively combines K one-to-one streams with a many-to-one stream to use information from multiple source contrasts.
- Each one-to-one stream is trained as a conditional GAN to generate the target image from one distinct source image.
- The many-to-one stream is trained as a conditional GAN using all source images concatenated at the input level.
- mustGAN fuses unique one-to-one feature maps with the shared many-to-one feature map, then trains a joint network to recover the target image.
- The fusion position is learned for task-specific optimization, and the joint-network architecture varies with that position.
2) Many-to-One Stream:
The many-to-one stream processes all source images jointly, after which mustGAN concatenates feature maps from every stream at a selected layer and uses a joint network for reconstruction.
- 2) Many-to-One Stream:: The additional (K + 1)th stream performs many-to-one synthesis from all source images.
- 2) Many-to-One Stream:: Its discriminator receives all source images and either the generated or real target image.
- 2) Many-to-One Stream:: The many-to-one generator predicts the target from source images concatenated at the input level.
- 2) Many-to-One Stream:: After stream training, mustGAN concatenates feature maps from the one-to-one and many-to-one streams at fusion layer i.
- 2) Many-to-One Stream:: A joint network reconstructs the target from fused features, with early, intermediate, and late fusion determined by the fusion position.
3) Joint Network:
The joint network fuses feature maps from the one-to-one and many-to-one streams at an adaptively selected position, then reconstructs the target image under adversarial and pixel-wise losses.
- Early Fusion: Early fusion concatenates one-to-one and many-to-one feature maps within the encoder before the joint network recovers the target image.
- Intermediate Fusion: Intermediate fusion places the fusion block within the residual network and passes the resulting maps to the joint network.
- Late Fusion: Late fusion places the fusion block within the decoder, combining stream feature maps before target-image recovery.
- The joint network is trained adversarially with a fixed convolutional discriminator and a loss combining adversarial and pixel-wise L1 terms.
B. Network Architecture
The one-to-one and many-to-one streams use matched generator and discriminator designs, while the joint network changes its architecture according to fusion-block location.
- The K one-to-one streams and many-to-one stream use identical generator and discriminator architectures.
- Each generator contains a three-layer convolutional encoder, nine-block residual network, and three-layer convolutional decoder.
- The discriminator is a five-layer convolutional network, with input channels varying across the one-to-one, many-to-one, and joint streams.
- Early, intermediate, and late fusion use progressively different joint-network compositions based on the fusion position.
C. Datasets
Demonstrations used normalized multi-contrast MRI from healthy subjects in IXI and glioma patients in ISLES, with separate training, validation, and test subjects.
- Each brain volume was normalized to a maximum intensity of 1 separately for each subject and MR contrast.
- IXI Dataset: The IXI dataset contained healthy-subject T1-, T2-, and PD-weighted images from 53 subjects.The split was 25 training, 10 validation, and 18 testing subjects.
- Approximately 100 artifact-free axial brain cross sections were manually selected from each subject in both datasets.
- ISLES Dataset: The ISLES dataset contained T1-, T2-, and FLAIR images from 56 glioma patients.The split was 25 training, 10 validation, and 21 testing subjects.
D. Network Training
Training proceeded in two phases: separate stream training followed by joint-network training, with cross-validation and grid search used to select fusion settings and training duration.
- First Phase: The first phase independently trained the one-to-one and many-to-one streams for 100 epochs using Adam-based conditional GAN training.
- First Phase: The stream objective weighted pixel-wise loss relative to adversarial loss by 100, with β1 = 0.5 and β2 = 0.999.
- Second Phase: The second phase trained the joint network after fusion and evaluated fusion-block positions for each synthesis task.
- Second Phase: One-fold cross-validation with grid search selected both fusion position and the number of training epochs for the joint network.
E. Competing Methods
The evaluation compared mustGAN with pGAN variants across IXI and ISLES datasets, using task-specific fusion-layer optimization and PSNR/SSIM measurements. Optimal fusion placement differed by dataset, with measurable effects on PSNR.
- Evaluation setup: Two public datasets, IXI and ISLES, were used to evaluate mustGAN against pGANone and pGANmany.All methods were trained and tested on the same samples, and performance was assessed using SSIM and PSNR after intensity normalization.
- Evaluation setup: IXI experiments evaluated three many-to-one synthesis directions and 14 candidate fusion-block positions for mustGAN.The compared directions were T2, PD → T1; T1, PD → T2; and T1, T2 → PD.
- IXI fusion optimization: Late fusion mostly improved IXI synthesis, with optimal fusion at layers 12, 14, and 12 for T1, T2, and PD synthesis, respectively.The PSNR gaps between the highest- and lowest-performing fusion models were 0.794 dB, 1.057 dB, and 0.655 dB for the three tasks.
- ISLES fusion optimization: Intermediate fusion mostly improved ISLES synthesis, with optimal fusion at layers 8, 6, and 6 for T1, T2, and FLAIR synthesis, respectively.The corresponding highest-to-lowest PSNR differences were 0.621 dB, 0.499 dB, and 0.623 dB.
- Dataset-dependent fusion: Fusion placement varied between datasets: IXI favored decoder fusion, whereas ISLES favored residual-block fusion.The paper relates this difference to IXI’s higher-SNR images and ISLES’s more limited resolution, emphasizing fine-detail versus global-structure recovery.
B. Demonstrations Against Competing Methods
Across healthy-subject IXI images and glioma-patient ISLES images, mustGAN produced visually sharper and less noisy synthesis than competing methods. Quantitative comparisons likewise favored mustGAN across tasks, with limited nonsignificant SSIM exceptions.
- IXI demonstrations: On IXI, mustGAN outperformed pGANone and pGANmany in all reported cases except SSIM for T1 and T2 synthesis.The reported differences were significant at p < 0.05 except for those SSIM comparisons, where p > 0.05.
- IXI demonstrations: On IXI, mustGAN achieved 0.94% higher SSIM and 1.27 dB higher PSNR than pGANmany on average.pGANmany exceeded pGANone in PSNR for 18.00% of test samples, while mustGAN reduced that proportion to 7.81%.
- IXI demonstrations: IXI visualizations showed lower white-matter noise and sharper gray-matter boundaries for mustGAN than for competing methods.The figure compares source images, ground-truth targets, mustGAN, pGANmany, and pGANone across T1, T2, and PD synthesis tasks.
- ISLES demonstrations: On ISLES, mustGAN again outperformed the competing methods in all cases except SSIM for T1 synthesis.The reported significance was p < 0.05 except for the T1 SSIM comparison, where p > 0.05.
- ISLES demonstrations: On ISLES, mustGAN achieved 1.29% higher SSIM and 0.84 dB higher PSNR than pGANmany on average.pGANone exceeded pGANmany in PSNR for 27.6% of test samples, while mustGAN reduced that proportion to 13.2%.
- ISLES demonstrations: ISLES visualizations likewise showed lower white-matter noise and sharper gray-matter boundaries for mustGAN than for competing methods.The comparisons covered T1, T2, and FLAIR synthesis in high- and low-grade glioma patients.
V. DISCUSSION
mustGAN combines complementary information from individual source contrasts with shared information across multiple sources, using an adaptively positioned fusion block. Across brain MRI synthesis tasks, it is reported to produce higher-quality images than isolated one-to-one or many-to-one methods, while relying on registered, paired source-target data.
- Method: mustGAN aggregates unique source-contrast features through one-to-one streams with shared multi-source features through a many-to-one stream.The streams are combined with a fusion block.
- Results: mustGAN recovered higher-quality images with reduced noise and improved sharpness compared with isolated one-to-one or many-to-one methods.The discussion reports this outcome across synthesis tasks on brain MRI datasets from normal subjects and glioma patients.
- Method: Adaptive fusion placement distinguishes mustGAN from Multimodal, whose fusion block is fixed at the initial decoder layer.The fusion position is modified to optimize task-specific performance.
- Scope and limitations: The current implementation assumes paired source-target images from the same subjects and requires spatial registration before synthesis.The paper identifies end-to-end registration and alternatives for unpaired data as future or potential extensions.