Swin Transformer for Fast MRI

1. Introduction

• Alternating shifted-window attention limits self-attention to local windows while enabling cross-window relationships across consecutive transformer layers.The introduction presents this as a way to balance computational complexity with the scale and high resolution of computer-vision images.Source: §1 · p. 6
• SwinMR combines parallel imaging with a Swin Transformer for fast CS-MRI reconstruction.The introduction identifies this model as the paper’s main contribution.Source: §1 · p. 6
• The proposed sensitivity-map-based multi-channel loss was reported to preserve more reconstruction textures and details.Source: §1 · p. 6
• The evaluation plan includes ablations and comparisons, tests across undersampling trajectories and noise conditions, and a downstream segmentation experiment on reconstructed images.A pretrained segmentation network is used to assess segmentation scores.Source: §1 · pp. 6–7

2.1. Classic Model-Based CS-MRI Reconstruction

• Traditional CS-MRI formulates recovery from undersampled k-space as an optimization problem to improve spatial information while reducing artifacts.The supplied passage introduces the optimization problem but does not include its equation.Source: §2.1 · p. 7
• The sparsifying transform Φ may be a discrete wavelet transform, gradient operator, or dictionary-based transform.The regularizer R(·) acts on sparsity, with examples including the l1-norm and l0-norm; λ balances the two terms.Source: §2.1 · p. 7
• The optimization can be solved using nonlinear methods such as gradient-based algorithms and variable-splitting methods.Source: §2.1 · p. 7
• Manually designed regularization may require long reconstruction times to achieve better reconstruction quality.Source: §2.1 · p. 7
• Manually selected transforms can introduce artifacts: total-variation regularization may create staircase artifacts, while tight wavelet frames may produce blocky artifacts.The passage also notes that total variation removes noise and preserves sharp edges, and that tight wavelet frames increase reconstruction efficiency.Source: §2.1 · p. 7

2.2. CNN-based Fast MRI Reconstruction

• Deep CNNs were first applied to CS-MRI to directly map down-sampled reconstruction images to fully sampled images, prompting further quality-improvement networks.CNNs were used as feature extractors to learn this mapping.Source: §2.2 · p. 7
• Hybrid methods mimicked iterative classic algorithms: ADMM-Net unfolded ADMM into network blocks, while another approach iteratively incorporated a CNN prior.The CNN prior reconstructs from lower-quality images and takes the zero-filled reconstruction as input.Source: §2.2 · pp. 7–8
• Conditional GANs enhanced reconstruction quality through alternating generator-discriminator optimization, learning reconstructions from undersampled zero-filled images.After training, the generator maps the zero-filled input to reconstructed images.Source: §2.2 · p. 8
• GAN variants addressed generator, discriminator, loss, regularization, stability, and attention, while DAGAN combined a modified U-Net with competitive training and VGG transfer learning.PIDDGAN incorporated edge information, and transfer learning improved generalizability for networks trained on small datasets.Source: §2.2 · pp. 8–9
• CNN methods improved reconstruction quality and efficiency over classical methods, but convolutional locality limited performance and motivated SwinMR.The paper identifies the local sensitivity of convolution as the limitation motivating its Swin transformer-based method.Source: §2.2 · p. 9

2.3.1. Overall Architecture

• Undersampled, optionally noise-corrupted k-space data are converted to zero-filled inputs, which SwinMR reconstructs with a residual connection; sensitivity maps support multi-channel output formation.Ground-truth coil images are combined into a single-channel image using root sum square, while sensitivity maps are estimated with ESPIRiT and used to form multi-channel reconstructions.Source: §2.3.1 · p. 9, Figure 2 · p. 10

2.3.2. Input Module and Output Module

• A Conv2D input module maps the single-channel image to C feature channels, and a Conv2D output module maps FEM features back to a single-channel image.The modules preserve the spatial dimensions H × W while changing channel count.Source: §2.3.2 · pp. 11–12

2.3.3. Feature Extraction Module

• The feature extraction module cascades residual Swin transformer blocks followed by a Conv2D, with each block using patch embedding, Q Swin transformer layers, patch unembedding, and a residual connection.A residual connection also links the feature extraction module to the input module output.Source: Figure 3 · p. 11, §2.3.3 · p. 12

2.3.4. Swin Transformer Layer

• STLs alternate window and shifted-window attention instead of whole-image MSA, reducing attention computation while enabling communication between windows.Each STL also includes layer normalization, an MLP, and residual connections.Source: §2.3.4 · p. 13

2.3.5. Loss Function

• The total loss combines sensitivity-map-based multi-channel pixel-wise and frequency Charbonnier losses with a VGG perceptual loss.The authors introduce the sensitivity-map-based loss to improve reconstruction quality and preserve textures and details; they state that the VGG loss optimizes perceptual quality.Source: §2.3.5 · pp. 16–17

3.1. Datasets

• Experiments used the 67-case, 12-channel CC dataset and multimodal BraTS17 volumes, with 40/7/20 CC and 235/20/30 BraTS17 cases for training, validation, and testing.The study sampled 100 central slices per case for CC and for BraTS17 training.Source: §3.1 · pp. 17–18

3.2. Implementation Detail

• SwinMR was trained for 100k Adam steps with random flips and rotations; PI used multi-channel data and sensitivity maps, whereas nPI used single-channel data without sensitivity maps.Training used two NVIDIA RTX 3090 GPUs; testing used an RTX 3090 GPU or Intel Core i9-10980XE CPU.Source: §3.2 · p. 18

3.3. Evaluation Methods

• Reconstruction evaluation used SSIM, PSNR, and FID; tumour segmentation used IoU and Dice, while model size and computational cost were measured by #PARAMs and MACs.The paper describes lower FID as indicating more perceptual results and calculates MACs using a 1 × 1 × 256 × 256 input array.Source: §3.3 · pp. 18–19

3.4. Comparisons with Other Methods

• SwinMR had higher MACs than CNN- and GAN-based methods but a smaller model size; its computational cost was higher than other CNN-based models.Table 1 defines inference time as the average over ten inferences on an Intel Core i9-10980XE CPU or NVIDIA RTX 3090 GPU.Source: §3.4 · p. 19, Table 1 · p. 20
• SwinMR showed superiority to other methods in overall reconstruction quality and edge information in the CC comparison.Figure 4 presents reconstructed images, Sobel edge information, and 10× absolute differences from GT images.Source: §3.4 · p. 19, Figure 4 · p. 20
• SwinMR (nPI) achieved the highest SSIM and PSNR, while SwinMR (PI) achieved the best FID under Gaussian 1D 30% masking on CC.Table 1 reports mean (std) comparison results and paired t-tests against the two SwinMR variants.Source: §3.4 · p. 19, Table 1 · p. 20

3.5. Experiments on Masks

• The mask experiment compared SwinMR (PI), DAGAN, and zero-filling on CC data using Gaussian 1D 10%, 30%, and 50%, radial 10%, and spiral 10% trajectories.SSIM, PSNR, and FID were evaluated.Source: §3.5 · pp. 20–21
• SwinMR (PI) achieved higher reconstruction quality than DAGAN across undersampling trajectories, especially at 10%, with significantly higher PSNR and SSIM (p < 0.05).The comparison used the Calgary-Campinas dataset.Source: §3.5 · p. 21, Figure 5 · p. 22
• Mask-condition visual analyses showed reconstructed images, Sobel edge information, and 10× absolute differences from ground truth for the five trajectories.The figures include zero-filled, DAGAN, and SwinMR (PI) reconstructions.Source: §3.5 · p. 21, Figure 6 · p. 23, Figure 7 · p. 24

3.6. Experiments on Noise

• The noise experiment tested NL20%, NL30%, NL50%, NL70%, and NL80% after Gaussian 1D 30% undersampling, comparing SwinMR (PI), DAGAN, and zero-filling on CC data.The study evaluated SSIM, PSNR, and FID.Source: §3.6 · p. 26
• Noise-condition visual analyses showed reconstructed images, Sobel edge information, and 10× absolute differences from ground truth across the five noise levels.The figures include zero-filled, DAGAN, and SwinMR (PI) reconstructions.Source: §3.6 · p. 26, Figure 10 · p. 28, Figure 11 · p. 29
• SwinMR maintained better reconstruction quality than DAGAN under noise, with improvement clearer at high noise; PSNR and SSIM were significantly higher (p < 0.05).Noise was tested after Gaussian 1D 30% undersampling in k-space.Source: §3.6 · p. 26, Figure 9 · p. 27

3.7. Ablation Experiments on the Patch Number and Channel Number

• Larger patches improve reconstruction quality and produce faster, lower training loss, but increase computational cost; the study used patch number 96.The reported patch-number results include SSIM, PSNR, FID, and training loss.Source: §3.7 · p. 31, Figure 13 · p. 32
• Changing channel number yields no significant differences in SSIM, PSNR, or FID, while larger channel numbers produce faster, lower training loss; the study used 180 channels.The channel-number ablation reports SSIM, PSNR, FID, and training loss.Source: §3.7 · p. 31, Figure 13 · p. 32
• SwinMR (PI) tends to have lower FID but lower SSIM and PSNR than SwinMR (nPI).Source: §3.7 · p. 31

3.8. Ablation Experiments on the Loss Function

• The loss ablation compared PFP (pixel-wise, frequency, and perceptual), PP (pixel-wise and perceptual), PF (pixel-wise and frequency), and P (pixel-wise only).The study aimed to isolate the effect of each loss term.Source: §3.8 · p. 34
• Frequency loss generally improves SSIM/PSNR and lowers FID for SwinMR (PI), but for SwinMR (nPI) improves SSIM/PSNR while scarcely affecting FID.The reported comparisons are PFP vs PP and PF vs P; the study says frequency loss benefits reconstruction quality metrics in most cases.Source: §3.8 · pp. 34–35
• Perceptual loss tends to lower SSIM/PSNR but substantially improve FID for SwinMR (PI), while improving FID with little SSIM/PSNR change for SwinMR (nPI).In most cases, perceptual loss benefits FID but harms SSIM/PSNR when using multi-channel data.Source: §3.8 · p. 35

3.9. Downstream Task Experiments: Brain Segmentation Experiments on BraTS17 Dataset

• The downstream segmentation model is an open-access multi-modal brain-tumour network with a U-Net-based architecture, residual blocks, strided-convolution downsampling, and squeeze-and-excitation channel attention.Channel attention is applied to concatenated multi-level features.Source: §3.9 · p. 37
• BraTS17 segmentation was evaluated on ground-truth, SwinMR-reconstructed, and zero-filled images using a pre-trained network after training SwinMR separately on FLAIR, T1, T1CE, and T2.The segmentation network was trained on the four required modalities: FLAIR, T1, T1CE, and T2.Source: §3.9 · p. 37
• Reconstructed-image IoU and Dice scores improve over zero-filled images and are much closer to ground-truth scores; under the Gaussian 1D 30% mask, their distributions are not significantly different from ground truth (p > 0.05).The comparison uses a pre-trained segmentation network.Source: §3.9 · p. 37

T2 T1CE T1 FLAIR

• Figure 18 shows BraTS17 FLAIR, T1, T1CE, and T2 ground truths, zero-filled inputs, and SwinMR reconstructions under Gaussian 1D 10% and 30% masks.The figure arranges ground truth, zero-filled, and reconstructed images in separate rows for both mask rates.Source: Figure 18 · p. 38
• Figure 19 displays BraTS17 segmentation references and predictions from ground-truth, zero-filled, and SwinMR-reconstructed images under Gaussian 1D 10% and 30% masks.The visual marks whole tumour in blue, enhancing tumour in red, and tumour core in green.Source: Figure 19 · p. 39
• Table 4 reports BraTS17 reconstruction comparisons between SwinMR reconstructions and zero-filled images using PSNR, SSIM, and FID, summarized as mean (std).The caption identifies Gaussian 1D masks at 10% and 30%.Source: Table 4 · p. 40
• Tables 5 and 6 report BraTS17 segmentation IoU and Dice scores, respectively, for predictions using ground-truth, reconstructed, and zero-filled images under both mask rates.Scores are given as median/mean [Q1,Q3], with significance markers defined by Mann–Whitney tests against ground-truth images.Source: Table 5 · p. 40, Table 6 · p. 41

Discussion motivation

• SwinMR uses shifted-window attention to expand beyond CNNs’ limited receptive fields without relying on whole-image attention, which the authors considered redundant for MRI detail recovery [36].They argue that global attention may neglect local texture and detail, whereas shifted windows offer a receptive-field trade-off for computer vision tasks.Source: §4 · pp. 41–42

Discussion results

• SwinMR achieved the highest SSIM/PSNR and lowest FID among the compared CNN- and GAN-based models, with cerebellum details well preserved in zoomed reconstructions.The visual comparison also reports better reconstruction quality, particularly in the zoom-in area.Source: §4 · p. 42
• SwinMR (PI) showed more detail and texture, while SwinMR (nPI) appeared smoother; PI had better FID, whereas nPI had better SSIM/PSNR.The authors suggest the metric principles explain this split: PSNR is per-pixel, SSIM measures structural similarity, and FID compares high-dimensional InceptionV3 representations; added detail may improve visual similarity even if inaccurate.Source: §4 · pp. 42–43

Discussion limitations

• Transformer methods incurred higher computational cost than CNN- and GAN-based methods, and fixed window sizes and heavy computation remained limitations; multi-scale windows and lightweight transformers were proposed for future work.The authors note that (S)W-MSA complexity is proportional to the input’s H × W, with inference time reported at 256 × 256; training used random crops to reduce processing time.Source: §4 · pp. 43–44

Discussion results

• Across tests with different undersampling masks and noise levels, SwinMR outperformed DAGAN, while the evaluation metrics changed as expected with the conditions.Source: §4 · p. 43

Discussion ablations

• Increasing patch number improved reconstruction quality with diminishing gains but increased computational complexity, motivating the choice of 96; channel-number changes showed no positive metric correlation, and 180 was selected empirically.The authors assumed metrics had saturated across tested channel numbers and used SwinIR’s default setting.Source: §4 · p. 43
• Frequency loss improved SSIM and PSNR, while perceptual VGG loss improved FID, consistent with their respective emphasis on reconstruction fidelity and perception.Pixel-wise and frequency losses primarily constrained fidelity; VGG loss focused on perception.Source: §4 · p. 43

• SwinMR is a novel parallel-imaging-coupled Swin transformer model for fast multi-channel MRI reconstruction.Source: §5 · p. 44
• SwinMR outperformed benchmark CNN-based and GAN-based MRI reconstruction methods.Source: §5 · p. 44
• SwinMR showed excellent robustness across different undersampling trajectories and noise conditions.Source: §5 · p. 44