PaperlayerView exact revision on arXiv

Source-linked AI summary

One Model to Synthesize Them All: Multi-contrast Multi-scale Transformer for Missing Data Imputation

Jiang Liu, Srivathsa Pasumarthi, Ben Duffy, Enhao Gong, Keshav Datta, Greg Zaharchuk

arXiv:2204.13738v3eess.IVcs.CV

TL;DR

MRI contrasts provide complementary information, but their availability varies among patients, creating a missing-contrast imputation problem. MMT uses a sequence-to-sequence Transformer to synthesize missing contrasts from available inputs, and experiments report better performance than competing methods across datasets and scenarios.

  • Problem

    MRI contrast availability varies among patients, although the contrasts provide complementary information about anatomy and pathology.

    Source: §I · p. 1

  • Method

    MMT formulates imputation as sequence-to-sequence prediction and uses a multi-scale Transformer with multi-contrast shifted-window attention.

    Source: §I · p. 2, B. Multi-contrast Shifted Window-based… · p. 4

  • Results

    MMT significantly outperforms MILR and MMGAN across all metrics and scenarios on IXI and BraTS (Wilcoxon p < 0.005).

    Source: E. Evaluation Results · p. 7

  • Takeaways & Limitations

    Tumor-segmentation experiments show diagnostic equivalence between MMT-imputed and ground-truth images.

    Source: §V · p. 12

    The method requires registered, same-orientation inputs; unreliable registration leads to poor performance.

    Source: §V · p. 13

Abstract

from arXiv · show

Multi-contrast magnetic resonance imaging (MRI) is widely used in clinical practice as each contrast provides complementary information. However, the availability of each imaging contrast may vary amongst patients, which poses challenges to radiologists and automated image analysis algorithms. A general approach for tackling this problem is missing data imputation, which aims to synthesize the missing contrasts from existing ones. While several convolutional neural networks (CNN) based algorithms have been proposed, they suffer from the fundamental limitations of CNN models, such as the requirement for fixed numbers of input and output channels, the inability to capture long-range dependencies, and the lack of interpretability. In this work, we formulate missing data imputation as a sequence-to-sequence learning problem and propose a multi-contrast multi-scale Transformer (MMT), which can take any subset of input contrasts and synthesize those that are missing. MMT consists of a multi-scale Transformer encoder that builds hierarchical representations of inputs combined with a multi-scale Transformer decoder that generates the outputs in a coarse-to-fine fashion. The proposed multi-contrast Swin Transformer blocks can efficiently capture intra- and inter-contrast dependencies for accurate image synthesis. Moreover, MMT is inherently interpretable as it allows us to understand the importance of each input contrast in different regions by analyzing the in-built attention maps of Transformer blocks in the decoder. Extensive experiments on two large-scale multi-contrast MRI datasets demonstrate that MMT outperforms the state-of-the-art methods quantitatively and qualitatively.

I. INTRODUCTION

MRI contrasts provide complementary information, but missing or corrupted scans make synthesis from available contrasts important. The paper frames this task as sequence-to-sequence prediction and introduces MMT, which synthesizes missing contrasts from variable input subsets.

I. INTRODUCTION

  • Missing contrasts can reflect variable availability, corruption, or acquisition protocols, while each MRI contrast highlights different anatomy or pathology.Imputation aims to synthesize missing contrasts from existing ones and can avoid rescanning or modifying pipelines for missing contrasts.Source: §I · p. 1
  • CNN-based imputation methods require fixed input and output channels, struggle with long-range dependencies, and lack interpretability.The introduction notes that CNN approaches may require up to 2P − 2 models for P contrasts in the worst case.Source: §I · p. 2
  • MMT formulates imputation as sequence-to-sequence prediction, mapping variable-length combinations of available contrasts to missing contrasts with one Transformer model.This formulation accommodates arbitrary missing subsets, unlike image-to-image translation from a fixed set of inputs.Source: §I · p. 2
  • MMT combines hierarchical multi-scale encoding with coarse-to-fine decoding, while M-Swin attention captures intra- and inter-contrast dependencies.Decoder attention scores indicate which input contrasts and regions contribute information to each output pixel.Source: §I · p. 2
  • Experiments on two large-scale multi-contrast MRI datasets show MMT's quantitative and qualitative superiority over state-of-the-art methods.Source: §I · p. 3

A. MRI image synthesis

  • MRI image synthesis includes one-to-one, one-to-many, many-to-one, and many-to-many tasks, while prior work also explored multi-input synthesis.The related-work overview describes traditional patch regression, atlas-based, and MR-physics approaches alongside CNN-based methods.Source: A. MRI image synthesis · p. 3

B. Transformer

The section frames Transformers as a way to handle long-range dependencies and describes missing-contrast imputation as a challenge across possible input-output configurations. MMT is presented as a flexible, multiscale and interpretable approach to that task.

B. Transformer

  • Transformers encode input sequences and generate output sequences, and their ability to capture long-range dependencies has supported their use in computer vision and medical imaging.The section names prior MRI applications including PTNet, T2Net, SLATER, and ResViT.Source: B. Transformer · p. 3

C. Missing data imputation

  • Existing imputation methods cover only some configurations or rely on fusion or pre-imputation, while CNN-based approaches have fixed-channel, long-range-dependency, and interpretability limitations.ResViT uses a Transformer bottleneck but remains largely CNN-based, pre-imputes missing contrasts, and lacks interpretability.Source: C. Missing data imputation · p. 3

A. Overview

  • MMT’s overview represents available contrasts as encoder inputs and describes synthesis of the remaining contrasts through decoder contrast queries.The decoder reasons about input-target contrast relationships to produce the output sequence.Source: A. Overview · p. 3
  • MMT accepts arbitrary subsets of contrasts, builds hierarchical multiscale representations, and uses decoder attention maps to indicate which input contrasts and regions inform output pixels.The overview also describes CNN-based discriminators used during training to improve image quality.Source: A. Overview · pp. 3–4

B. Multi-contrast Shifted Window-based Attention

M-Swin computes attention within local windows spanning contrasts, capturing within- and between-contrast dependencies. Successive regular and shifted window partitions connect neighboring windows while keeping computation tractable for high-resolution images.

B. Multi-contrast Shifted Window-based Attention

  • Local cross-contrast windows let each pixel attend to same-contrast and other-contrast pixels, modeling intra- and inter-contrast dependencies.The approach is described as generalizing Swin’s window-based attention, which processes single-modality data.Source: B. Multi-contrast Shifted Window-based… · p. 4
  • Shifted window partitioning in successive blocks connects neighboring non-overlapping windows from the previous layer.Fig. 2 depicts regular partitioning in one layer followed by shifted partitioning in the next.Source: B. Multi-contrast Shifted Window-based… · p. 4, Figure 2 · p. 4
  • Local window attention reduces computational complexity relative to global attention and can operate regardless of the number of contrasts.This supports arbitrary-subset input and missing-contrast generation with one model.Source: B. Multi-contrast Shifted Window-based… · p. 4

1) Overall architecture:

The encoder jointly processes multicontrast inputs into hierarchical representations at multiple scales. Its U-Net-like path uses patch operations for downsampling and upsampling, while M-Swin blocks provide regular- and shifted-window attention.

1) Overall architecture:

  • The encoder embeds each input image separately, partitions encoded images into 4×4 patches, and treats the resulting tokens as inputs to attention.The encoder is designed similarly to a U-Net to produce multiscale representations.Source: 1) Overall architecture · p. 4
  • Patch merging concatenates neighboring 2×2 patches to halve spatial resolution and double feature dimension; patch expanding reverses those changes, and skip concatenations help retain spatial information.The encoder outputs representations at multiple scales along its upsampling path.Source: 1) Overall architecture · p. 4

2) M-Swin encoder block:

  • M-Swin encoder blocks replace standard multi-head attention with regular- and shifted-window multi-contrast attention to jointly encode inputs and capture inter- and intra-contrast dependencies.The encoder architecture produces hierarchical representations at multiple scales.Source: 2) M-Swin encoder block · p. 4, Figure 3 · p. 5
  • Learnable contrast encodings inject contrast-specific information, and relative position bias is added during attention computation.The block also includes LayerNorm and a two-layer GELU perceptron.Source: 2) M-Swin encoder block · p. 4

D. MMT Decoder

The MMT decoder uses contrast queries and multiscale decoding to generate target images from encoder outputs. Its cross-contrast attention also makes information flow interpretable, while shallow CNN blocks and discriminators complement the Transformer.

1) Overall Architecture:

  • Contrast queries guide the decoder as M-Swin blocks progressively decode encoder outputs across scales to generate target images coarse to fine.The decoder restores image resolution before an image decoding block produces the final image.Source: 1) Overall Architecture · p. 5, Figure 4 · p. 5

2) M-Swin Decoder block:

  • Decoder cross-attention uses input-contrast features as keys and values and the target-contrast feature as the query, with scores measuring regional information flow.These attention scores provide a quantitative view of which input contrasts and regions contribute information to synthesis.Source: 2) M-Swin Decoder block · p. 5

E. CNN Blocks

  • Shallow CNN blocks encode images before the Transformer and decode its outputs; separate encoding and decoding blocks are used for each contrast.The paper evaluates this design with an ablation study.Source: E. CNN Blocks · pp. 5–6

F. Discriminator

  • Multi-scale CNN-based discriminators guide MMT toward realistic details and correct global structure, with a separate discriminator for each contrast.The discriminator approach is adopted from.Source: F. Discriminator · p. 6

G. Loss functions

MMT training combines synthesis, reconstruction, and adversarial losses. The synthesis term targets accurate missing-contrast generation, while reconstruction regularizes the encoder to preserve input information.

G. Loss functions

  • The MMT training objective comprises synthesis, reconstruction, and adversarial loss components.The supplied passages describe the synthesis and reconstruction terms in this section.Source: G. Loss functions · p. 6

1) Synthesis Loss:

  • Synthesis loss measures pixel-wise similarity and trains MMT to synthesize missing contrasts accurately from available contrasts.The available passages name the loss and its purpose but do not provide its displayed formula.Source: 1) Synthesis Loss · p. 6

2) Reconstruction Loss:

  • Reconstruction loss is the L1 distance between input and reconstructed images, regularizing the encoder to preserve input information.The decoder is queried with the contrast queries of the input contrasts for reconstruction.Source: 2) Reconstruction Loss · p. 6

3) Adversarial Loss:

Adversarial training uses an LSGAN objective to encourage realistic synthesized images. Label smoothing stabilizes training, and adversarial learning aims to match synthetic and real image distributions for each contrast.

3) Adversarial Loss:

  • The adversarial loss uses an LSGAN objective, with fake and real labels and a discriminator for each contrast.The loss encourages MMT to generate realistic images that fool the discriminators.Source: 3) Adversarial Loss · p. 6
  • Label smoothing stabilizes training by sampling labels from uniform distributions instead of fixing them at 0 or 1.Source: 3) Adversarial Loss · p. 6
  • Adversarial learning between MMT and the discriminators forces synthetic-image distributions to match real-image distributions for each contrast.Source: 3) Adversarial Loss · p. 6

4) Overall Loss:

The generator loss is a weighted sum of three terms, and evaluation uses IXI and BraTS brain MRI data. The experiments compare methods across missing-contrast scenarios using image-quality and perceptual metrics.

4) Overall Loss:

  • The generator loss is the weighted sum of synthesis, reconstruction, and adversarial losses, with a separate weight for each term.Source: 4) Overall Loss · p. 6

A. Datasets

  • IXI includes 577 healthy-subject scans with T1, T2, and PD contrasts, while BraTS includes 1,251 patient scans with four brain MRI contrasts.The datasets use distinct preprocessing and train/validation/test splits; two BraTS cases were excluded for poor image quality.Source: A. Datasets · p. 6

B. Evaluation Settings

  • Evaluation uses PSNR, SSIM, and LPIPS to compare MMT with MILR and MMGAN for single-missing and random-missing contrast scenarios.In the random-missing scenario, one to K − 1 contrasts may be missing.Source: B. Evaluation Settings · p. 6

C. Implementation Details

  • MMT was trained for 100 epochs using AdamW, with available contrasts randomly sampled at each iteration; window sizes were 8 × 8 for IXI and 6 × 5 for BraTS.Experiments used 8 Nvidia Tesla V100 GPUs; loss weights were λr = 5, λs = 20 and λadv = 0.1, chosen based on validation-set qualitative and quantitative results.Source: C. Implementation Details · pp. 6–7

D. Handling variable inputs and outputs

MMT accepts variable combinations of available contrasts and synthesizes the missing ones. It outperforms the compared CNN methods across the evaluated datasets and scenarios, and its imputed images support tumor segmentation.

D. Handling variable inputs and outputs

E. Evaluation Results

  • MMT significantly outperforms MILR and MMGAN across all metrics and scenarios on both IXI and BraTS.Source: E. Evaluation Results · p. 7
  • MMT random models reliably synthesize missing contrasts across input combinations, with performance improving as more input contrasts are available.Source: E. Evaluation Results · p. 7

F. Tumor Segmentation

  • MMT achieves Dice scores above 0.9 in most single-missing-contrast segmentation cases, while imputed T1Gd is more challenging than other contrasts.Source: F. Tumor Segmentation · p. 8
  • Dice(maskori, maskmmt) > Dice(maskori, maskmiss), indicating that MMT-imputed sequences can supplement information for tumor segmentation.Source: F. Tumor Segmentation · p. 8

G. Radiomics Feature Analysis

The radiomics analysis compares features from ground-truth and synthetic zero-gad T1Gd tumor regions. The reported feature correlations support the diagnostic utility of the synthetic images.

G. Radiomics Feature Analysis

H. Ablation Studies

The ablations show that CNN image encoding and decoding blocks improve perceptual and visual quality, while cross-contrast attention contributes to interpretable attention maps. Decoder visualizations also show contrast-specific and coarse-to-fine synthesis behavior.

1) CNN Blocks:

  • Removing the CNN image encoding and decoding blocks raises LPIPS and produces blocky outputs, while the blocks make outputs more continuous and visually closer to ground truth.Source: 1) CNN Blocks · p. 10, Figure 11 · p. 12

2) Cross-contrast Attention:

  • Without cross-contrast attention, attention maps activate randomly across contrasts rather than on lesion areas, although quantitative performance is almost similar.Source: 2) Cross-contrast Attention · p. 10

3) Adversarial Loss:

I. Attention Score Analysis

  • For T1Gd synthesis, decoder attention mostly uses T1 outside tumors but draws more information from T2 and FLAIR in tumor regions.Source: §I · p. 11
  • Decoder attention features begin coarse and progressively become finer, focusing on the lesion while synthesizing T1Gd.Source: §I · p. 11
  • On IXI, PD contributes about 70% of attention information for T1 or T2 synthesis; on BraTS, T1/T1Gd contribute about 50% and T2/FLAIR about 40% for each other.Source: §I · p. 12

V. DISCUSSIONS AND CONCLUSION

MMT outperforms state-of-the-art methods on IXI and BraTS and shows diagnostic equivalence to ground-truth images on tumor segmentation. The discussion highlights perceptual-quality and attention findings alongside clinical possibilities and important limits.

V. DISCUSSIONS AND CONCLUSION

  • MMT achieves better quantitative and qualitative performance than state-of-the-art methods on IXI and BraTS, with diagnostic equivalence on tumor segmentation.Source: §V · p. 12
  • With cross-contrast attention, synthesized T1Gd is better than without it, especially in the red-circled region; the attention heat map is also described as more explainable.Source: Figure 12 · p. 12
  • LPIPS is consistently superior for MMT even when SSIM/PSNR differences from competing methods are not significant.The authors interpret this as better overall perceptual quality despite similar pixel-based quality metrics.Source: §V · p. 12
  • Potential applications include avoiding gadolinium administration in some patients, synthesizing redundant contrasts, and replacing missing or corrupted sequences to avoid extra scans.These are described as potential applications, including zero-gad synthesis and addressing scan quality or motion issues.Source: §V · pp. 12–13
  • Future extensions proposed include cross-modality synthesis and segmentation, including joint training of synthesis and segmentation.Source: §V · p. 13
  • The work requires registered multi-contrast inputs in the same orientation; registration may be time-consuming, and unreliable registration can lead to poor performance.The authors also call for reader studies and multi-center evaluation, and note memory, parameter, and training-sample demands.Source: §V · p. 13
Loading 2204.13738v3…