PaperlayerView exact revision on arXiv

Source-linked AI summary

Missing MRI Pulse Sequence Synthesis using Multi-Modal Generative Adversarial Network

Anmol Sharma, Ghassan Hamarneh

arXiv:1904.12200v3eess.IVcs.AIcs.CVcs.LGstat.ML

TL;DR

Missing MRI sequences matter because downstream analysis pipelines may require particular sequences as input. The paper proposes a multi-modal GAN that combines available sequences to synthesize missing ones in a single forward pass. It reports outperforming competing unimodal and multi-modal methods in the evaluated settings.

  • Problem

    Missing MRI pulse sequences matter because many downstream analysis pipelines assume a specific set of sequences as input.

    Source: §I · p. 1

  • Method

    MM-GAN combines information from available sequences to synthesize all missing sequences in a single forward pass.

    Source: §VII · p. 12

  • Results

    MM-GAN outperformed REPLICA and MM-Synthesis in multi-input T2flair synthesis, while retaining high PSNR and SSIM in harder missing-sequence scenarios.

    Source: §VII · p. 12

  • Takeaways & Limitations

    The synthesis process accommodates any combination of available and missing sequences, with runtime constant as the number of missing sequences varies.

    Source: §VII · p. 12

    The method has difficulty synthesizing the enhancing subregion in T1c, which contains tumor information not present in other sequences.

    Source: §VII · p. 12

Abstract

from arXiv · show

Magnetic resonance imaging (MRI) is being increasingly utilized to assess, diagnose, and plan treatment for a variety of diseases. The ability to visualize tissue in varied contrasts in the form of MR pulse sequences in a single scan provides valuable insights to physicians, as well as enabling automated systems performing downstream analysis. However many issues like prohibitive scan time, image corruption, different acquisition protocols, or allergies to certain contrast materials may hinder the process of acquiring multiple sequences for a patient. This poses challenges to both physicians and automated systems since complementary information provided by the missing sequences is lost. In this paper, we propose a variant of generative adversarial network (GAN) capable of leveraging redundant information contained within multiple available sequences in order to generate one or more missing sequences for a patient scan. The proposed network is designed as a multi-input, multi-output network which combines information from all the available pulse sequences, implicitly infers which sequences are missing, and synthesizes the missing ones in a single forward pass. We demonstrate and validate our method on two brain MRI datasets each with four sequences, and show the applicability of the proposed method in simultaneously synthesizing all missing sequences in any possible scenario where either one, two, or three of the four sequences may be missing. We compare our approach with competing unimodal and multi-modal methods, and show that we outperform both quantitatively and qualitatively.

I. INTRODUCTION … B. Multimodal Synthesis

The paper motivates missing-sequence synthesis as a way to preserve complementary diagnostic information and downstream usability, and presents MM-GAN as many-input, many-output synthesis for all missing-sequence combinations.

I. INTRODUCTION

  • MRI pulse sequences provide different tissue contrasts, and particular sequences support distinct diagnostic tasks, including delineating tumor edema and enhancing regions or detecting white matter hyperintensities.T1 and T2flair delineate edema, T1c demarcates enhancing tumor regions, and T2flair supports vascular-dementia diagnosis.Source: §I · p. 1
  • Missing or corrupted sequences can result from variable protocols, artifacts, incorrect settings, contrast-agent allergies, or limited scan time, disrupting downstream pipelines that expect specific inputs.The paper notes that many brain-MRI segmentation methods may fail when required sequences are absent –.Source: §I · p. 1
  • Synthesizing missing sequences can avoid difficult pipeline modifications or impractical repeat scans, which entail expense, delays, registration, and possible anatomical change between scans.The paper highlights rapidly changing abnormalities such as glioblastoma as a reason anatomy may differ between scans.Source: §I · p. 1
  • MM-GAN combines all available pulse sequences to synthesize missing ones in one forward pass, using four-channel input and output while allowing generalization to other sequence counts.Missing input channels are zero-filled; the four target sequences are T1, T1c, T2, and T2flair,,.Source: §I · pp. 1–2

II. RELATED WORK

  • The related-work review distinguishes unimodal one-to-one synthesis from multimodal methods that use multiple inputs to synthesize one or multiple sequences.It surveys MR pulse-sequence synthesis methods [2],, –.Source: §II · p. 2

A. Unimodal Synthesis

  • Unimodal synthesis spans atlas and patch-based intensity mapping,,, and learned CRF, deep-network, random-forest, and GAN approaches,,,,,.These methods synthesize a target sequence from a single input sequence.Source: A. Unimodal Synthesis · p. 2

B. Multimodal Synthesis

  • Prior multimodal work includes regression, encoder-decoder, and GAN approaches,,,, but the cited methods focus on a single synthesized output.Some analysis methods instead tolerate missing sequences [3],, but the paper describes a trade-off between missing-input robustness and performance when all sequences are available.Source: B. Multimodal Synthesis · pp. 2–3
  • The authors claim MM-GAN is the first method demonstrated to synthesize multiple missing sequences from multiple inputs across every combination of missing sequences.Earlier multimodal methods synthesized only one output sequence; the paper notes that prior many-to-many scalability was not empirically demonstrated.Source: B. Multimodal Synthesis · p. 3

III. CONTRIBUTIONS … 1) Generator:

The paper presents MM-GAN for synthesizing missing MRI pulse sequences from available sequences in one forward pass, and reports improved quantitative performance from implicit conditioning.

III. CONTRIBUTIONS

  • MM-GAN synthesizes any combination of missing sequences from any combination of available sequences in one forward pass, using one model without tuning or retraining.This avoids the training overhead of multiple models required by unimodal and multi-input single-output approaches.Source: §III · p. 3
  • The method outperforms the unimodal methods REPLICA and pGAN and the multi-input single-output method MM-Synthesis, and establishes BraTS2018 benchmarks across scenarios.The contributions passage describes these benchmarks as covering a complete set of scenarios.Source: §III · p. 3
  • Implicit conditioning combines zero-imputed missing inputs, sequence-selective generator loss, and sequence-selective discrimination, improving quantitative synthesis performance; the work also introduces curriculum-based GAN training.The curriculum varies the difficulty of examples presented during training.Source: §III · p. 3

A. Background

  • A GAN pairs a generator that produces data with a discriminator that distinguishes real from generated data; Pix2Pix conditions generation on an input image and uses patchwise real/fake predictions.The background also describes conditional GANs as conditioning generation on a random vector and class label.Source: A. Background · p. 3

T ′ Generator

  • Figure 1 depicts a UNet generator and PatchGAN discriminator, with copy/drop blocks and generator L1 loss applied only to synthesized missing sequences.The discriminator receives real and generated images and outputs a four-channel patchwise real/fake prediction.Source: Figure 1 · p. 4

B. Proposed Method

  • MM-GAN is a Pix2Pix variant that leverages all available sequences to synthesize missing MR pulse sequences in a single forward pass.The paper introduces this model before detailing its architecture.Source: B. Proposed Method · p. 4

1) Generator:

  • The generator takes 256 × 256 axial slices with four fixed channels for T1, T2, T1c, and T2flair, zero-imputing channels for missing sequences.The four-channel network processes 3D scans sequentially, one 2D axial slice at a time.Source: 1) Generator · p. 4
  • Generator training combines L1 reconstruction on missing sequences with adversarial L2 loss; its PatchGAN discriminator classifies each 16 × 16 patch.The adversarial target tensor is set to ones to encourage generated sequences to be judged real; L1 is used to limit blurring.Source: 1) Generator · pp. 4–5, 2) Discriminator · p. 5

2) Discriminator:

The discriminator uses imputed generator outputs and real inputs to focus discrimination on missing sequences; this is part of implicit conditioning alongside zero-input imputation and selective generator loss.

2) Discriminator:

  • Replacing outputs for already-present sequences with real inputs before concatenating Xi and Xr focuses discrimination on missing outputs and helps stabilize training.The imputed generator output Xi retains synthesized sequences only where inputs were missing, allowing the discriminator to distinguish those from their real counterparts.Source: 2) Discriminator · p. 5, 3) Implicit conditioning · p. 5
  • D predicts a 16×16 patch with four sequence channels, labeled 0 for missing sequences and 1 for present sequences across the whole image, not patchwise.Each channel corresponds to one sequence, and each 256×256 image is divided into 16×16 patches.Source: 2) Discriminator · p. 5
  • The discriminator uses an L2 loss between its output and the target, making the objective equivalent to a least-squares GAN.Source: 2) Discriminator · p. 5
  • Implicit conditioning combines zero imputation for missing generator inputs, selective generator loss on missing outputs, and selective discrimination; it outperforms generating and penalizing all sequences.Selective generator loss ignores sequences already present in the input, while selective discrimination prevents the discriminator from overpowering the generator.Source: 3) Implicit conditioning · p. 5

3) Implicit conditioning: … D. Training and Implementation Details

This block describes MM-GAN’s curriculum, evaluation datasets and preprocessing, benchmark protocols, and training implementation.

3) Implicit conditioning:

  • The authors hypothesized that presenting easier cases when supervision is ample could help MM-GAN learn before harder cases are introduced.The passage frames curriculum learning as a way to let the network adapt without diverging as more difficult cases are added.Source: 4) Curriculum learning · pp. 5–6

4) Curriculum learning:

  • Curriculum training presented scenarios with one missing sequence first, then two, then three, before showing all scenarios with uniform probability after 30 epochs.The curriculum increased scenario difficulty every 10 epochs through epoch 30, then continued with uniform sampling until epoch 60 in MIMO experiments.Source: 4) Curriculum learning · p. 6, D. Training and Implementation Details · p. 7

V. EXPERIMENTAL SETUP; A. Datasets

  • Evaluation uses ISLES2015 SISS (22 training patients selected from 28 training cases) and BraTS2018 HGG/LGG cohorts, with 195/65 training, 5 each validation, and 10/5 testing patients.The datasets provide four sequences each: T1, T2, DWI, and T2flair for ISLES2015; T1, T2, T1c, and T2flair for BraTS2018.Source: A. Datasets · p. 6, §V · p. 6

B. Preprocessing

  • Scans were mean-normalized, cropped to a shared brain bounding box, and resized to 256 × 256 slices for compatibility with the encoder-decoder generator.Mean normalization was chosen over min-max normalization because pathological outlier intensities could compress the intensity range and artificially lower the L1 score.Source: B. Preprocessing · p. 6

C. Benchmark Methods

  • Comparisons cover pGAN/Pix2Pix one-to-one and REPLICA/MM-Synthesis many-to-one; MISO tests seven ISLES2015 T2flair-missing scenarios, while MIMO tests all 14 valid BraTS2018 scenarios.For many-to-one synthesis, REPLICA required seven models, while MM-Synthesis and MM-GAN used one network each to cover the seven scenarios.Source: C. Benchmark Methods · pp. 6–7

D. Training and Implementation Details

  • Training used Adam with learning rate η = 0.0002, β1 = 0.5, and β2 = 0.999, with generator-loss weight λ = 0.9 and alternating single generator and discriminator steps.The discriminator loss was multiplied by 0.5 to slow discriminator learning relative to the generator.Source: D. Training and Implementation Details · p. 7
  • Experiments examined one-to-one versus many-to-one synthesis, MISO comparisons, component ablations, and MIMO synthesis; MISO used batches of 2 for 30 epochs, while MIMO used batches of 4 for 60 epochs.Ablations tested average, noise, or zero imputation and the effects of curriculum learning and implicit conditioning.Source: D. Training and Implementation Details · p. 7
  • MIMO used the original PatchGAN discriminator and a ReLU output, whereas data-limited MISO used a smaller noisy-input discriminator and a linear output; inference averaged 0.1536 ± 0.0070 seconds per patient.The smaller MISO discriminator and noise inputs were used to prevent the discriminator from overpowering the generator.Source: D. Training and Implementation Details · p. 7

E. Evaluation Metrics · VI. RESULTS AND DISCUSSION · A. Single-Input VS Multi-Input Synthesis

The evaluation uses MSE, PSNR, and SSIM, and reports that MI-GAN’s multiple inputs improve missing-sequence synthesis over single-input methods while reducing the number of required models.

E. Evaluation Metrics

  • The evaluation reports MSE, PSNR, and SSIM, generally normalizing ground truth and synthesized images to [0, 1], but omitting re-normalization for Section VI-B.PSNR accounts for MSE and maximum supported intensity; SSIM compares image statistics to capture perceived quality.Source: E. Evaluation Metrics · pp. 7–8

VI. RESULTS AND DISCUSSION

  • Experiments compare unimodal one-to-one and multi-input single-output methods, then report multi-input multi-output benchmarks; the single-versus-multiple-input test synthesizes T1 and T2 from BraTS2018 data.The test uses 70 training patients and 5 test patients, with Wilcoxon signed-rank tests and significant p-values reported at p < 0.05.Source: §VI · p. 8, A. Single-Input VS Multi-Input Synthesis · p. 8

A. Single-Input VS Multi-Input Synthesis

B. T2flair Synthesis (MISO) · C. Multimodal Synthesis (MIMO)

MM-GAN synthesizes missing T2flair across varied input scenarios and extends this to multimodal synthesis across all 14 valid scenarios. Results report a T2flair benchmark advantage, high-fidelity synthesis, and graceful degradation as more sequences are missing.

B. T2flair Synthesis (MISO)

  • MM-GAN's MSE (0.226±0.046) beat REPLICA (0.271±0.10) in all seven scenarios and MM-Synthesis (0.236±0.08) in four of seven.MM-GAN also had lower MSE standard deviation than both methods in all but one scenario (T2 missing).Source: B. T2flair Synthesis (MISO) · p. 9, Table II · p. 10
  • Qualitatively, MM-GAN produced sharper T2flair with fewer blurring artifacts, preserved high-frequency detail, and followed the real sequence's intensity distribution.The reported comparisons with MM-Synthesis draw on qualitative results from its original paper.Source: B. T2flair Synthesis (MISO) · pp. 9–10

B. T2flair Synthesis (MISO); C. Multimodal Synthesis (MIMO)

  • MISO curriculum learning was ineffective because more inputs did not ensure easier synthesis (T1 and T2 MSE 0.2541 vs DW-only MSE 0.2109), whereas implicit conditioning stabilized MIMO training.The authors attribute MISO's result to some sequences being more informative than others; implicit conditioning made the MIMO discriminator's task harder and helped prevent it from overpowering the generator.Source: B. T2flair Synthesis (MISO) · p. 10, C. Multimodal Synthesis (MIMO) · p. 11

C. Multimodal Synthesis (MIMO)

  • For one-sequence synthesis, MM-GAN trained on all 14 valid scenarios to synthesize a missing sequence from any available-sequence combination achieved SSIM >0.90 in most cases (except one in LGG) and PSNR >22.0.Source: C. Multimodal Synthesis (MIMO) · p. 10
  • In tumor synthesis, T1c provides unique enhancing-region information, T1 and T2flair improve edema synthesis, and T2 helps synthesize the tumor mass when combined with other inputs.MM-GAN preserves grey-white matter boundary detail while recreating tumor regions from available sequences; tumor synthesis depends on which sequences are present.Source: C. Multimodal Synthesis (MIMO) · p. 10
  • MM-GAN recreated a cut-off frontal-lobe region and high-frequency detail in T2flair using available higher-resolution sequences.The authors suggest resolution upscaling as a possible use but leave its investigation for future work.Source: C. Multimodal Synthesis (MIMO) · pp. 10–11, Figure 4 · p. 11
  • In HGG, performance drops for two missing sequences versus one were 27.1% in MSE, 2.7% in PSNR, and 0.7% in SSIM; for three missing, drops were 39.1%, 7.2%, and 2.2%.Compared with two missing sequences, three missing yielded drops of 29.3% in MSE, 4.6% in PSNR, and 1.5% in SSIM; the method retained high PSNR and SSIM in harder scenarios.Source: C. Multimodal Synthesis (MIMO) · p. 11, Table V · p. 11
  • For HGG synthesis, inputs ranked T1c>T2>T1 for T2flair, T2>T1>T2flair for T1c, T1c>T1>T2flair for T2, and T1c>T2>T2flair for T1.Removing T1c caused the synthesized T2flair to miss the necrotic tumor region, while removing T2 increased blurring in synthesized T1c.Source: C. Multimodal Synthesis (MIMO) · p. 11

VII. CONCLUSION

MM-GAN synthesizes missing MRI pulse sequences from available sequences in a single forward pass, and the conclusion reports comparative results, sequence-specific limitations, and directions for future work.

VII. CONCLUSION

  • The multi-input, multi-output MM-GAN handles any combination of available and missing sequences in one forward pass, with runtime constant in the number of missing sequences.The conclusion contrasts this with prior unimodal or partially multimodal approaches; MI-GAN also outperformed unimodal pGAN in all three metrics.Source: §VII · p. 12
  • MM-GAN outperformed REPLICA and MM-Synthesis in multi-input T2flair synthesis, while BraTS2018 experiments established a benchmark in MSE, PSNR, and SSIM.The paper also reports objectively sharper and more accurate results and shows that the method can fill missing details and improve synthesized sequences.Source: §VII · p. 12
  • MM-GAN performance degrades as more sequences are missing, but it maintains high PSNR and SSIM even in harder scenarios.The conclusion also reports that more available sequences allow better synthesis of missing ones.Source: §VII · p. 12
  • The method has difficulty synthesizing the T1c enhancing subregion because that sequence contains tumor-enhancement information absent from the other sequences.More generally, MRI sequences contain both redundant and unique information, which challenges unimodal and multimodal synthesis methods.Source: §VII · p. 12
  • For T2flair synthesis, DWI input achieved MSE 0.2109 versus T1 input at MSE 0.2813 and T2 input at MSE 0.2799, demonstrating sequence-dependent accuracy.The authors note that faithful synthesis depends on having another sequence that captures similar characteristics.Source: §VII · p. 12
  • Future work could extend MM-GAN from 2D to 2.5D or 3D, investigate upscaling scans with missing areas, and test its use in downstream segmentation pipelines.The authors suggest this pipeline placement as another way to make segmentation robust to missing pulse sequences, alongside HeMIS and PIMMS.Source: §VII · p. 12

I. VALIDATION OF KEY COMPONENTS · A. Imputation Value and Curriculum Learning

The validation tests missing-sequence imputation strategies with curriculum learning (CL) and random sampling (RS); zero imputation with CL is selected for the final benchmark training, with the best reported MSE and PSNR.

A. Imputation Value and Curriculum Learning

B. Effect of Implicit Conditioning (IC)

Implicit conditioning (IC) combines zero imputation with selective generator loss and discriminator input to focus training on missing sequences. On the LGG cohort, IC improved all three reported metrics over training without IC.

B. Effect of Implicit Conditioning (IC)

  • IC combines zero imputation, selective loss computation in G, and selective discrimination in D to train the network to synthesize missing sequences without penalizing inaccuracies in available ones.Source: B. Effect of Implicit Conditioning (IC) · p. 14
  • The non-IC comparison retained zero imputation for architectural reasons but penalized synthesis of all four sequences and fed the discriminator the full generator output.Source: B. Effect of Implicit Conditioning (IC) · p. 14
  • With IC, the LGG-trained MM-GAN outperformed non-IC training on MSE (0.0095 vs 0.0100), PSNR (24.2487 vs 23.6626), and SSIM (0.8946 vs 0.8796; p < 0.01).Networks in the final benchmark were trained with IC enabled.Source: B. Effect of Implicit Conditioning (IC) · p. 14
  • Without IC, the generator had to synthesize every sequence and the discriminator quickly overpowered it; selective discrimination instead supported stable training by making real and fake sequences harder to distinguish.The authors attribute the performance gap to the generator's limited capacity and the absence of selective loss computation in the non-IC model.Source: B. Effect of Implicit Conditioning (IC) · p. 14

II. RECONSTRUCTION ERROR IN OTHER PLANES

The study compared reconstruction-error distributions across axial, sagittal, and coronal planes for five LGG test patients. The tests suggested no significant differences between axial and either other plane.

II. RECONSTRUCTION ERROR IN OTHER PLANES

  • Mean squared reconstruction errors were calculated in axial, sagittal, and coronal planes for five test patients in the LGG cohort.The comparison examined whether using 2D axial slices in training and testing was associated with inconsistencies in other planes.Source: §II · p. 14
  • The study used Mann–Whitney U tests at a 0.05 confidence threshold to compare axial-versus-sagittal and axial-versus-coronal error distributions.The test compares unpaired sample distributions without assuming they were drawn from a normal distribution.Source: §II · pp. 14–15
  • Across five LGG test patients, Mann–Whitney tests suggested no significant reconstruction-error differences between axial and sagittal or coronal planes.The study reported that the null hypothesis could not be rejected for either comparison.Source: §II · pp. 14–15

T1 T2 T1c T2f MSE PSNR SSIM MSE PSNR SSIM

The reported mean±std metrics summarize two result blocks, while accompanying passages enumerate sequence-availability configurations and describe statistical testing on five LGG test patients.

Related papers

Loading 1904.12200v3…