PaperlayerView exact revision on arXiv

Source-linked AI summary

Swin Transformer for Fast MRI

Jiahao Huang, Yingying Fang, Yinzhe Wu, Huanjun Wu, Zhifan Gao, Yang Li, Javier Del Ser, Jun Xia, Guang Yang

arXiv:2201.03230v2eess.IVcs.AIcs.CVcs.LG

TL;DR

Long MRI scans can cause discomfort and motion-related artefacts, creating a need for fast reconstruction from undersampled measurements. SwinMR couples parallel imaging with shifted-window Swin transformers and sensitivity-map-based multi-channel loss. It outperformed benchmark CNN- and GAN-based methods and remained robust across undersampling trajectories and noise conditions, while computational cost remains a limitation.

  • Problem

    Long acquisition times make high-quality MRI uncomfortable and susceptible to voluntary and physiological motion artefacts, motivating faster reconstruction from undersampled k-space data.

  • Method

    SwinMR is a parallel-imaging-coupled Swin transformer model for fast multi-channel MRI reconstruction using shifted-window attention and sensitivity maps.

  • Results

    SwinMR outperformed benchmark CNN- and GAN-based MRI reconstruction methods and showed robustness across different undersampling trajectories and noise levels.

  • Takeaways & Limitations

    Shifted-window attention provides a larger receptive field than CNNs while avoiding whole-image attention, supporting detailed MRI reconstruction within the reported experiments.

  • Takeaways & Limitations

    Fixed window sizes and substantial transformer computational cost remain limitations of the method.

Abstract

from arXiv · show

Magnetic resonance imaging (MRI) is an important non-invasive clinical tool that can produce high-resolution and reproducible images. However, a long scanning time is required for high-quality MR images, which leads to exhaustion and discomfort of patients, inducing more artefacts due to voluntary movements of the patients and involuntary physiological movements. To accelerate the scanning process, methods by k-space undersampling and deep learning based reconstruction have been popularised. This work introduced SwinMR, a novel Swin transformer based method for fast MRI reconstruction. The whole network consisted of an input module (IM), a feature extraction module (FEM) and an output module (OM). The IM and OM were 2D convolutional layers and the FEM was composed of a cascaded of residual Swin transformer blocks (RSTBs) and 2D convolutional layers. The RSTB consisted of a series of Swin transformer layers (STLs). The shifted windows multi-head self-attention (W-MSA/SW-MSA) of STL was performed in shifted windows rather than the multi-head self-attention (MSA) of the original transformer in the whole image space. A novel multi-channel loss was proposed by using the sensitivity maps, which was proved to reserve more textures and details. We performed a series of comparative studies and ablation studies in the Calgary-Campinas public brain MR dataset and conducted a downstream segmentation experiment in the Multi-modal Brain Tumour Segmentation Challenge 2017 dataset. The results demonstrate our SwinMR achieved high-quality reconstruction compared with other benchmark methods, and it shows great robustness with different undersampling masks, under noise interruption and on different datasets. The code is publicly available at https://github.com/ayanglab/SwinMR.

1. Introduction

Fast MRI reconstruction addresses the long acquisition times and motion-related artefacts of high-quality MRI. SwinMR combines parallel imaging with a shifted-window Swin transformer and multi-channel loss, supported by comparative, ablation, robustness, and segmentation experiments.

  • Long MRI acquisitions can cause patient discomfort and motion-related artefacts, motivating accelerated reconstruction from undersampled k-space measurements.
  • Classical parallel imaging and compressed sensing reduce k-space measurements, but iterative or manually regularized reconstruction can prolong processing and introduce artefacts.
  • CNN-based MRI reconstruction is efficient but limited by convolution’s local sensitivity, restricted receptive field, computationally costly large kernels, and risks from excessive depth.
  • SwinMR is a parallel-imaging-coupled Swin transformer model for fast CS-MRI reconstruction, using shifted-window attention rather than whole-image attention.
  • A sensitivity-map-based multi-channel loss was proposed to preserve more textures and details, alongside experiments testing masks, noise, ablations, and downstream segmentation.

2.1. Classic Model-Based CS-MRI Reconstruction

Classic model-based CS-MRI reconstructs images from limited k-space measurements using sparsity-promoting optimization. Its reconstruction quality and speed depend on manually selected transforms, regularization, and iterative solvers.

  • Traditional CS-MRI reconstructs an image from a small portion of k-space by imposing a sparsifying transform and solving a constrained optimization problem.
  • Common sparsifying transforms include discrete wavelets, gradient operators, and dictionary-based transforms, with l1- or l0-norm regularization balancing reconstruction terms.
  • Gradient-based and variable-splitting solvers can require long reconstruction times, while hand-crafted regularization may introduce staircase or blocky artefacts.

2.2. CNN-based Fast MRI Reconstruction

CNN-based methods replaced lengthy hand-crafted reconstruction procedures with learned mappings and improved MRI reconstruction efficiency. Their local convolutional sensitivity motivated SwinMR, while GANs further targeted reconstruction quality.

  • Deep CNNs were applied to learn a direct mapping from downsampled reconstruction images to fully sampled reconstruction images, relieving hand-crafted regularization and long reconstruction time.
  • Some approaches unfolded optimization algorithms into network blocks or incorporated CNN priors iteratively to approximate classic compressed-sensing solutions.
  • Conditional GANs used alternating generator-discriminator training to enhance MR reconstruction quality from undersampled zero-filled images.
  • GAN variants addressed generator, discriminator, loss, regularization, stability, attention, edge information, and small-dataset generalizability concerns.
  • CNN-based MR reconstruction improved reconstruction quality and efficiency over classical methods, but convolution’s local sensitivity limited performance.

2.3. SwinMR: Swin Transformer for MRI Reconstruction

SwinMR reconstructs undersampled MRI through convolutional input/output modules and a residual Swin-transformer feature extractor, with sensitivity-map-based multi-channel supervision. Windowed and shifted-window attention restricts computation locally while enabling communication across windows.

  • Overall architecture: SwinMR converts undersampled zero-filled images into reconstructed MR images using an input module, feature extraction module, output module, and residual connections.The input and output modules use Conv2D layers, while the feature extraction module cascades RSTBs followed by a Conv2D.
  • Dataflow: The dataflow uses RSS to form single-channel ground truth, k-space undersampling and noise interruption to create inputs, and sensitivity maps to recover multi-channel reconstructions.Sensitivity maps are estimated by ESPIRiT, and reconstructed single-channel images are multiplied pixel-wise by those maps.
  • Residual Swin transformer blocks: Each residual Swin transformer block combines patch embedding, cascaded Swin transformer layers, patch unembedding, Conv2D processing, and a residual connection.Each STL contains layer normalization, attention, an MLP, and residual connections.
  • Windowed attention: Alternating regular and shifted windows let consecutive layers communicate across windows without applying self-attention over the whole image.Windowed attention reduces computational complexity relative to original whole-image MSA, while shifting prevents fixed-window separation.
  • Loss function: A multi-channel loss combines pixel-wise Charbonnier, frequency Charbonnier, and perceptual VGG losses.The sensitivity maps enter the pixel-wise loss, while the VGG loss uses feature differences from a VGG network.

3. Experiments and Results

Experiments evaluated SwinMR on the Calgary-Campinas dataset against established reconstruction methods using image-quality, segmentation, model-size, and computational-cost measures. SwinMR achieved the strongest reported reconstruction metrics and preserved image details, while requiring higher computational cost than CNN- and GAN-based methods.

  • Comparison experiments: The comparison used the Calgary-Campinas dataset and Gaussian 1D 30% undersampling against Deep ADMM Net, U-Net, DAGAN, PIDDGAN, and zero-filled images.Both single-channel and multi-channel SwinMR variants were included.
  • Quantitative results: SwinMR (nPI) achieved the highest SSIM and PSNR, while SwinMR (PI) achieved the best FID in the quantitative comparison.The evaluation also reported inference time, parameter count, and MACs.
  • Computational cost: SwinMR had higher computational cost than CNN- and GAN-based methods but a smaller model size in parameter count.The higher computational cost was associated with transformer attention over feature maps.
  • Visual results: SwinMR showed superior overall reconstruction quality and edge information compared with other methods.Visual comparisons included reconstructed images, Sobel-extracted edges, and amplified absolute differences from ground truth.

3.5. Experiments on Masks

SwinMR was tested across Cartesian and non-Cartesian undersampling trajectories and across increasing noise levels. It maintained higher reconstruction quality than DAGAN across masks and under noise, with clearer improvement at high noise levels.

  • Different undersampling masks: The mask experiment compared Gaussian 1D 10%, 30%, and 50%, radial 10%, and spiral 10% undersampling trajectories.SwinMR (PI), DAGAN, and zero-filled reconstruction were evaluated on the Calgary-Campinas dataset using SSIM, PSNR, and FID.
  • Different undersampling masks: SwinMR achieved higher reconstruction quality than DAGAN across different undersampling trajectories, especially with 10% undersampling masks.The comparison included quantitative metrics and image, edge, and absolute-difference visualizations.
  • Different undersampling masks: Mask visualizations compare zero-filled images, ground truth, DAGAN reconstructions, SwinMR reconstructions, and the corresponding undersampling patterns.PSNR and SSIM values are displayed in the sample-image panels, while edge information is extracted with the Sobel operator.
  • Noise robustness: Under noise interruption, SwinMR maintained better reconstruction quality than DAGAN, with clearer improvement at high noise levels.The study reported quantitative results alongside reconstructed images, edge information, and amplified absolute differences.

3.7. Ablation Experiments on the Patch Number and Channel Number

The ablations examine how patch and channel counts affect SwinMR reconstruction, training convergence, and the trade-off between quality and computational cost.

  • Patch number: Larger patch numbers improve reconstruction quality and make training loss converge faster and lower, but increase computational cost.The authors empirically selected a patch number of 96 for training.
  • Channel number: Changing the channel number produces no significant differences in SSIM, PSNR, or FID, although larger channel numbers yield faster and lower training-loss convergence.The empirically selected channel number was 180.
  • Multi-channel comparison: With multi-channel data and sensitivity maps, SwinMR tends to obtain lower FID but lower SSIM and PSNR than its single-channel variant.The variants are compared as SwinMR (PI) and SwinMR (nPI).
  • Evaluation: The ablation evaluates SSIM, PSNR, FID, and training loss across patch-number and channel-number settings.Figures 13–15 provide metric curves and reconstructed-image samples for these comparisons.

3.8. Ablation Experiments on the Loss Function

The loss ablation separates pixel-wise, frequency, and perceptual terms to assess their effects on SwinMR reconstruction metrics and downstream segmentation relevance.

  • Configurations: The ablation compares four loss configurations: pixel-wise, pixel-wise plus frequency, pixel-wise plus perceptual, and all three terms.The configurations are denoted P, PF, PP, and PFP.
  • Evaluation: The loss experiments report SSIM, PSNR, and FID and include reconstructed-image and edge-information samples.These evaluations are presented in Figures 16 and 17.
  • Frequency loss: Frequency loss tends to improve SSIM and PSNR and decrease FID for SwinMR (PI), while improving only SSIM and PSNR for SwinMR (nPI).The comparisons are PFP versus PP and PF versus P.
  • Perceptual loss: Perceptual loss substantially decreases FID for SwinMR (PI) but slightly decreases SSIM and PSNR, whereas for SwinMR (nPI) it improves FID with little SSIM/PSNR change.The comparisons are PFP versus PF and PP versus P.
  • Downstream evaluation: The downstream experiment applies a pretrained brain-tumour segmentation network to ground-truth, reconstructed, and zero-filled MR images.It uses four BraTS17 modalities: FLAIR, T1, T1CE, and T2.
  • Downstream evaluation: For Gaussian 1D 30% undersampling, reconstructed-image IoU and Dice distributions are closer to ground truth than zero-filled images and are not significantly different from ground truth.The Mann–Whitney test reports p > 0.05 for reconstructed versus ground-truth distributions.

T2 T1CE T1 FLAIR

The BraTS17 evaluation covers FLAIR, T1, T1CE, and T2 modalities, comparing ground truth, zero-filled inputs, reconstructions, and downstream segmentation outputs.

  • Reconstruction samples: Figure 18 organizes reconstruction samples by FLAIR, T1, T1CE, and T2 modality and compares ground truth, zero-filled, and reconstructed images.The zero-filled and reconstructed images use Gaussian 1D 10% and 30% masks.
  • Segmentation samples: Figure 19 compares segmentation reference, ground-truth predictions, zero-filled predictions, and reconstructed-image predictions across undersampling masks.The columns include Gaussian 1D 10% and 30% conditions.
  • Quantitative reconstruction metrics: Table 4 reports PSNR, SSIM, and FID for SwinMR reconstructions and zero-filled images on BraTS17.Results are separated by modality and Gaussian 1D 10% or 30% mask.
  • Quantitative segmentation metrics: Tables 5 and 6 report segmentation IoU and Dice scores for whole tumour, enhancing tumour, and tumour core.The tables compare ground truth, reconstructed, and zero-filled images under Gaussian 1D 10% and 30% masks.

4. Discussion

SwinMR uses shifted-window attention to balance CNN locality and transformer context for MRI reconstruction, while experiments show strong reconstruction quality and robustness. The discussion also identifies metric trade-offs, computational costs, and remaining design limitations.

  • Motivation and architecture: SwinMR uses shifted-window attention to enlarge the receptive field beyond CNNs without applying self-attention across the whole image.Alternating normal and shifted windows enables communication between different windows.
  • Reconstruction quality: SwinMR achieved the highest SSIM/PSNR and lowest FID among the compared CNN-based and GAN-based MRI reconstruction models.The improvement was especially visible in zoomed regions where cerebellum details were preserved.
  • Metric and channel effects: Multi-channel SwinMR produced more details and textures with better FID, whereas single-channel SwinMR produced smoother images with better SSIM/PSNR.The authors relate this divergence to the different principles of the evaluation metrics.
  • Limitations: Transformer-based methods incur higher computational cost than CNN-based and GAN-based methods, and fixed attention windows remain a stated limitation.The authors suggest multi-scale windows and lightweight transformers as future directions.
  • Robustness: SwinMR outperformed DAGAN across tests using different undersampling masks and noise levels, demonstrating robustness under these conditions.The evaluation metrics changed as expected when masks and noise levels varied.
  • Ablation studies: Increasing patch number improved reconstruction quality until gradual saturation, but also increased computational complexity; the selected patch number was 96.Changing channel number was not positively correlated with evaluation metrics in this experiment.
  • Loss ablations: Frequency loss mainly improved fidelity-sensitive SSIM and PSNR, while perceptual loss positively affected perception-based FID.The proposed multi-channel loss uses sensitivity maps to preserve more textures and details.

5. Conclusion

The study developed SwinMR, a parallel-imaging-coupled Swin transformer model for fast multi-channel MRI reconstruction. It outperformed benchmark CNN- and GAN-based methods and showed robustness across undersampling trajectories and noise conditions.

  • SwinMR is a parallel-imaging-coupled Swin transformer model for fast multi-channel MRI reconstruction.
  • SwinMR outperformed benchmark CNN-based and GAN-based MRI reconstruction methods.
  • SwinMR showed excellent robustness across different undersampling trajectories with various noises.
Loading 2201.03230v2…