Source-linked AI summary
Missing MRI Pulse Sequence Synthesis using Multi-Modal Generative Adversarial Network
Anmol Sharma, Ghassan Hamarneh
TL;DR
Missing MRI sequences can result from scan corruption, protocol variation, limited time, or contrast-agent allergies, disrupting analyses that expect fixed inputs. The paper proposes MM-GAN, which combines available sequences and synthesizes one or more missing sequences in a single forward pass. Across brain-MRI experiments, it outperforms reported competing methods, while synthesis of T1c enhancement remains limited by information absent from other sequences.
Problem
MRI scans may have missing sequences, while downstream analysis pipelines often assume a fixed set of pulse-sequence inputs.
Method
MM-GAN is a multi-input, multi-output GAN that combines available sequences, implicitly conditions on missing ones, and synthesizes the missing sequences in one forward pass.
Results
MM-GAN outperforms unimodal and multimodal competing methods across the reported synthesis evaluations and remains robust in harder scenarios with more missing sequences.
Takeaways & Limitations
The method supports synthesizing arbitrary combinations of missing sequences with one trained model and constant runtime with respect to the number missing.
Takeaways & Limitations
T1c enhancement regions are difficult to synthesize because their highly specific information is absent from the other sequences.
Abstract
from arXiv · showhide
Magnetic resonance imaging (MRI) is being increasingly utilized to assess, diagnose, and plan treatment for a variety of diseases. The ability to visualize tissue in varied contrasts in the form of MR pulse sequences in a single scan provides valuable insights to physicians, as well as enabling automated systems performing downstream analysis. However many issues like prohibitive scan time, image corruption, different acquisition protocols, or allergies to certain contrast materials may hinder the process of acquiring multiple sequences for a patient. This poses challenges to both physicians and automated systems since complementary information provided by the missing sequences is lost. In this paper, we propose a variant of generative adversarial network (GAN) capable of leveraging redundant information contained within multiple available sequences in order to generate one or more missing sequences for a patient scan. The proposed network is designed as a multi-input, multi-output network which combines information from all the available pulse sequences, implicitly infers which sequences are missing, and synthesizes the missing ones in a single forward pass. We demonstrate and validate our method on two brain MRI datasets each with four sequences, and show the applicability of the proposed method in simultaneously synthesizing all missing sequences in any possible scenario where either one, two, or three of the four sequences may be missing. We compare our approach with competing unimodal and multi-modal methods, and show that we outperform both quantitatively and qualitatively.
I. INTRODUCTION
MRI combines multiple pulse sequences that provide complementary tissue contrasts, but sequences may be unavailable because of acquisition constraints and varying clinical protocols. The paper motivates synthesizing missing sequences so existing downstream analyses can continue without rescanning or redesigning pipelines.
- I. INTRODUCTION: MRI pulse sequences provide varied tissue contrasts and spatial resolutions within a scan, including T1, T2, T1c, and T2flair.Different sequences can be especially informative for edema, tumor enhancement, and white matter hyperintensities.
- I. INTRODUCTION: Clinical scans may lack routinely acquired sequences because of corruption, artifacts, incorrect settings, contrast-agent allergies, limited scan time, or protocol variation.These causes produce varying sets of available sequences across patients.
- I. INTRODUCTION: Missing sequences challenge downstream pipelines because many brain-MRI segmentation methods assume a fixed set of pulse sequences and may fail with missing inputs.Changing those pipelines to handle missing sequences can also degrade performance.
- I. INTRODUCTION: Repeating a scan to recover a sequence is impractical because acquisition is expensive, delays patients, requires registration, and may encounter rapidly changing anatomy.The paper therefore motivates recovering a missing sequence or estimate without rescanning or changing downstream pipelines.
- I. INTRODUCTION: Prior synthesis methods include unimodal one-to-one and multimodal many-to-one approaches, but the reviewed methods did not empirically demonstrate multi-output synthesis in one pass across varied missing-sequence scenarios.The paper positions its approach against this limitation in prior work.
III. CONTRIBUTIONS
The paper introduces MM-GAN, a multi-input, multi-output GAN that synthesizes missing MR sequences in one forward pass using available sequences and implicit conditioning. Its design combines fixed-channel imputation, selective losses and discrimination with UNet and PatchGAN components.
- III. CONTRIBUTIONS: MM-GAN is presented as an empirically validated multi-input, multi-output synthesizer that handles any combination of available and missing sequences without tuning or retraining.A single trained model produces target missing sequences in one forward pass.
- III. CONTRIBUTIONS: Implicit conditioning uses zero imputation for missing input channels, sequence-selective generator losses, and sequence-selective discrimination.These choices address fixed channel dimensions while focusing training on originally missing sequences.
- III. CONTRIBUTIONS: The generator is a UNet that receives a 4-channel 256 × 256 axial slice representing T1, T2, T1c, and T2flair.Missing channels are represented by zero images, while the network always outputs four channels.
- III. CONTRIBUTIONS: The generator combines an L1 reconstruction loss on missing sequences with an adversarial L2 loss that encourages generated outputs to appear real to the discriminator.The L1 term is computed only for sequences originally missing, and L1 is chosen to reduce excessive blurring relative to L2 reconstruction loss.
- III. CONTRIBUTIONS: The PatchGAN discriminator evaluates whether each N × N input patch is real or fake, encouraging accurate local as well as global image features.It receives a concatenation of the imputed generated output and the real sequences.
2) Discriminator:
The discriminator evaluates imputed multi-sequence outputs against real inputs while selectively distinguishing originally missing sequences. Its loss uses channel-specific supervision and an L2 objective, making it equivalent to a least-squares GAN.
- Discriminator: The discriminator receives a concatenation of imputed output Xi and real input Xr after restoring originally available sequences.Generated sequences that were originally present are dropped and replaced with their original inputs before discrimination.
- Discriminator: For each sequence channel, the target is 0.0 when the sequence is missing and 1.0 when it is available.The target assignment applies to the full 16×16 tensor for each channel rather than individual patches.
- Discriminator: The discriminator loss is computed with an L2 distance between its output and the target labels.The resulting objective is equivalent to a least-squares GAN.
- Discriminator: Implicit conditioning combines zero imputation, selective generator loss computation, and sequence-selective discrimination to handle varying missing-sequence patterns.These choices avoid penalizing reconstruction of sequences that were already provided.
3) Implicit conditioning:
The method uses implicit conditioning to support variable missing-sequence configurations with fixed-channel networks, and validates it through experiments on two four-sequence brain MRI datasets against unimodal and multimodal baselines.
- Implicit conditioning: Curriculum learning presents scenarios with one, two, then three missing sequences as easy, moderate, and hard cases.The ordering is intended to introduce increasingly difficult supervision during training.
- Experimental setup: Experiments use ISLES2015 and BraTS2018 brain MRI datasets, each organized around four pulse sequences.ISLES2015 uses T1, T2, DWI, and T2flair; BraTS2018 experiments include HGG and LGG cohorts.
- Preprocessing: Mean-intensity normalization is used because it is less sensitive than min-max normalization to extreme pathological voxel intensities.The authors report that min-max normalization could artificially lower L1 scores when most voxels were compressed near zero.
- Benchmark methods: The study compares MM-GAN with pGAN, REPLICA, MM-Synthesis, and Pix2Pix across unimodal, many-to-one, and many-to-many settings.The many-to-one comparison reproduces the ISLES2015 testbed with seven scenarios in which T2flair is synthesized.
- Training setup: The MIMO experiments use curriculum learning, while training otherwise uses fixed batch sizes and separate epoch schedules for MISO and MIMO.Models are trained for 30 epochs in MISO and 60 epochs in MIMO, with batch sizes of 2 and 4 respectively.
E. Evaluation Metrics
The evaluation combines pixelwise and perceptual metrics across single-input, multi-input, and multimodal synthesis experiments. Results reported in the supplied passages favor multi-input models and show that one MIMO model can cover all synthesis tasks.
- Evaluation Metrics: MSE, PSNR, and SSIM jointly measure pixelwise synthesis error and image quality aligned with human perception.MSE measures squared error, PSNR reflects signal-to-noise quality, and SSIM compares structural similarity.
- Single-Input VS Multi-Input Synthesis: Figure 2 compares T2flair synthesis across bit-string scenarios, with 1110 using T1, T2, and DW as available inputs.The caption states that this scenario most closely matches ground truth, with sharper tissue boundaries and less blurring than scenarios with fewer inputs.
- Single-Input VS Multi-Input Synthesis: Both MI-GAN variants outperform P2P and pGAN across MSE, PSNR, and SSIM.The comparison covers T1 synthesis from T2 and T2 synthesis from T1 using multiple available sequences.
- Single-Input VS Multi-Input Synthesis: 61.48% lower MSE, 20.29% higher PSNR, and 4.64% higher SSIM are reported for MI-GANT1 versus P2PT1.Each difference is reported with p < 0.05.
- Single-Input VS Multi-Input Synthesis: MI-GANT2 improves over pGANT2 by 2% in MSE, 2.64% in PSNR, and 0.77% in SSIM.The same passage also reports improvements over P2PT2 of 2%, 4.24%, and 1.64% for these metrics.
- Single-Input VS Multi-Input Synthesis: A many-to-many model requires one model for all synthesis tasks, compared with four many-to-one models and 12 one-to-one models when C = 4.The paper identifies this as a substantial reduction in training-time computational requirements.
B. T2flair Synthesis (MISO)
MM-GAN synthesizes missing T2flair sequences from varied combinations of available inputs and generally outperforms unimodal and multimodal comparators. Its performance depends on which informative sequences remain available, rather than simply on their number.
- T2flair synthesis: MM-GAN achieves MSE 0.226±0.046, outperforming REPLICA at 0.271±0.10 across all scenarios and MM-Synthesis at 0.236±0.08 in 4/7 scenarios.The model synthesizes T2flair with one, two, or three input sequences.
- Qualitative comparison: MM-GAN produces sharper T2flair images with fewer blurring artifacts than MM-Synthesis and REPLICA, while preserving high-frequency details.Its synthesized images also follow the intensity distribution of real T2flair sequences.
- Scenario dependence: The number of available sequences does not reliably predict synthesis quality because particular sequences can be more informative for a missing target.For example, T1 and T2 together yield MSE 0.2541 for T2flair, compared with 0.2109 using DW alone.
- Training: Implicit conditioning stabilized training despite different fake sequences appearing across iterations, helping the generator converge without observed unstable behavior.The discriminator was prevented from overpowering the generator during training.
- Unified inference: A single unified MM-GAN reduces the need to choose among separate unimodal synthesis networks during inference.Combining available sequences mitigates trade-offs caused by each individual sequence providing incomplete information, especially in tumorous regions.
VII. CONCLUSION
MM-GAN synthesizes missing MR pulse sequences by combining available sequences in a multi-input, multi-output model. It outperforms unimodal and multimodal competitors, while remaining limited in reconstructing T1c enhancing tumor regions.
- MM-GAN synthesizes all missing sequences from available sequences in a single forward pass, with runtime constant regardless of how many sequences are missing.
- MM-GAN outperformed REPLICA and MM-Synthesis for multi-input, single-output T2flair synthesis and produced sharper, more accurate results.
- The method had difficulty synthesizing the T1c enhancing subregion because that sequence contains highly specific information absent from the others.
- More available sequences enabled better synthesis of missing ones, reflecting the value of correlations across input sequences.
A. Imputation Value and Curriculum Learning
The study compares average, noise, and zero imputation under curriculum-learning and random-sampling strategies. Zero imputation with curriculum learning performed best for MSE and PSNR, while zero imputation with random sampling slightly led in SSIM.
- The experiments compared average, noise, and zero imputation together with curriculum learning and random sampling.
- Zero imputation with curriculum learning achieved the best reported MSE of 0.0095 and PSNR of 24.2487.
- Zero imputation with random sampling slightly outperformed curriculum learning in SSIM, reaching 0.8955.
- Noise imputation performed slightly worse than zero imputation but better than average imputation under both training strategies.
B. Effect of Implicit Conditioning (IC)
Implicit conditioning trains MM-GAN to synthesize only missing sequences while avoiding penalties for inaccurate reconstruction of available sequences. It improved all three reported metrics over training without implicit conditioning.
- Implicit conditioning combines zero imputation, selective generator loss computation, and selective discriminator input to focus training on missing sequences.
- The comparison retained zero imputation in the model without implicit conditioning because of architectural constraints.
- Implicit conditioning improved MSE to 0.0095 from 0.0100 compared with training without it.
- Implicit conditioning improved PSNR to 24.2487 from 23.6626 and SSIM to 0.8946 from 0.8796, with p < 0.01.
II. RECONSTRUCTION ERROR IN OTHER PLANES
The study evaluates reconstruction error across axial, sagittal, and coronal planes because the method trains and tests on 2D axial slices. Errors between axial and the other planes were compared statistically using the Mann-Whitney U test.
- Reconstruction errors were calculated for five LGG test patients across axial, sagittal, and coronal planes.
- Figure S1 compares qualitative T1 and T2 synthesis from P2P, pGAN, and MI-GAN, with red arrows marking tumor regions synthesized by MI-GAN.
- The Mann-Whitney U test compared error distributions between axial and sagittal planes and between axial and coronal planes using a 0.05 confidence threshold.
- Table SI encodes sequence absence or presence with bit strings from 0001 to 1110 for T1, T2, T1c, and T2flair.
- Table SII reports quantitative results for evaluating implicit-conditioning training.
T1 T2 T1c T2f MSE PSNR SSIM MSE PSNR SSIM
The table reports reconstruction results across missing-sequence configurations using MSE, PSNR, and SSIM, including mean±standard deviation values and statistical testing for five LGG test patients.
- Results: Configuration-specific results include MSE values from 0.0036±0.0016 to 0.0188±0.0035 in the displayed rows.The same rows report corresponding PSNR and SSIM values for the evaluated sequence combinations.
- Results: Mean±std results are reported for MSE, PSNR, and SSIM across the evaluated sequence configurations.The displayed aggregate values include 0.0100±0.0035, 23.6626±2.1386, and 0.8796±0.0337 for one metric grouping, and 0.0095±0.0039, 24.2487±2.4694, and 0.8946±0.0312 for another.
- Statistical analysis: The supplementary table reports Mann-Whitney U test p-values for axial-versus-coronal and axial-versus-sagittal comparisons on five LGG test patients.The statistical tests are conducted within the LGG cohort.